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SUMMARY 



aoDroal t f MPer deSCFibeS ^ reSults of a field evaluation of a new 
approach to the measurement of organizational productivity. This aoDroach in 

il ?^ tte 0hjeCti¥M Of ^ ^ (b) identifying ^L^^ n Sl 
eaters of how well the unit is meeting these objectives, and (c) develop^ fW 
STLST Tt'S^T Pe^ormance on the indicators and ^ Sib^on 
that those levels of the indicators make to overall effectiveness. 

The productivity measurement system was developed for five sections in 

~TZ:1 SUP f ata0 ° Perati0nal F -ce base' T^prXt vity me a" 
sures derived from the system were used as a basis for monthly feedback to tL 

^%eedb a y?° d r f ' 1Ve m ° nthS - Aft ° r «■ ^ settLg"aS to 
Ita g^e" ^ Finally ' «— ~ — — 5 the feedback 

tn M M,. ReSaI ! 8 016 System t0 be & y "y effective method of productivity 

measurement and enhancement. Its implementation was effective, unit pers2 
were cooperative m developing and using it, and it showed g oU psySn^Hc 
charactenstxcs Using the feedback that was produced by the system Sef n 

meT^ V 1 m Pr ^ Ctiv ^ of 5°* over baseline, across the f ive uuits 
#hen goal setting was added, the mean increase was 75% over baseline When 
~es were added, the mean increase was 7696 over baseline The" pos^iva 
effects lasted over time, and continued after the departure of the research team 
The incumbents and supervisors of the units evaluated the system very posit vely' 
After the research was completed, the units continued to use the system on Thefr 
own, and managers have requested that it be used in other units at L Tase 

This approach shows promise for use in much larger organizational units 

lent Worm^^f ^ alS ° applications in manag^ 

ment information systems, criterion development, test validation, measures of 
managerial performance, performance appraisal, and other situations where mul- 
tiple sources of data must be combined into an overall index or judgment 
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Efficiency vs. Effectiveness 



The first issue to be addressed is whether an efficiency or an ef- 
fectiveness approach should be used in measuring productivity. Both have been 
proposed and used* Efficiency is typically thought of as an output-to-input ratio. 
For example* monthly manufacturing output divided by manpower used to produce 
that output would be an efficiency measure* Effectiveness is usually discussed 
as the relationship of outputs to some standard or expectation* For example* 
monthly manufacturing output expressed as a percentage of the goal for that 
month would be an effectiveness measure* In addition, effectiveness usually in- 
cludes quality of the output as well as quantity. 

Efficiency is the more widely used of the two concepts because it is eas- 
ier to measure and standardize across organizations, industries* and nations 
(Norman & BahirL 1972)* When we hear that productivity growth in the United 
States has declined over the last 20 years (American Productivity Center* 1981), 
it is an efficiency ratio that is being quoted (i.e., price deflated gross national 
product divided by worker hours), Effectiveness is a much broader concept be- 
cause it includes other concepts such as standards, objectives of the organization, 
expectations of interested parties (e.g., shareholders, regulatory agencies, and 
customers), and the viability of the organization relative to its competition* As 
Mendelow (1983) argued* "The most efficient slide^rule manufacturer would be 
out of business today assuming it had not adapted its product line to meet the on- 
slaught of hand-held calculators 11 (p, 70), Proponents of the effectiveness con- 
cept argue that as complexity and ambiguity of the work increase, effectiveness 
measures become more important than efficiency (Balk, 1975). Effectiveness 
can also reflect the organization's bargaining position relative to its environment, 
whereas the efficiency concept does not, 

Some authors define productivity as a combination of efficiency and ef- 
fectiveness (Balk, 1975; Coulter, 1979; Hanes & Kriebel, 1978; National Center 
for Productivity and Quality of Working Life, 1978; Sibson, 1976; Tuttle, 1981), 
Typical formulations (Balk, 1975) are: 

Productivity - Efficiency + Effectiveness or 

Productivity = Output/Input + Output/ Standard 

Perspective Taken 

A second issue in the productivity literature is that although differing ap- 
proaches to productivity can be understood in terms of an efficiency, 
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effectiveness, or an integrated scheme, they can also be understood in terms of 
fte perspective, taken. That is, the approach to measuring productivity is 
determiaaed by the perspective of those doing the measuring. Tuttle (1983) 
suggeste^d five perspectives: 

1, Economist's Perspective. In this approach, productivity is output 
divided by associated inputs such as labor, capital, intermediate 
products purchased, and time. This approach is typically applied to 
very macro units such as whole industries or countries, 

2. Engineer's Perspective. In this approach, productivity is equated 
with the efficiency of the operation, based on a comparison of en- . 
ergy as the input and useful work as the output. This approach 
would typically be used with one organization, or a part of the or- 
ganization. In addition, it would typically focus on the equip- 
ment/hardware aspects of the organization, 

3. Accountant's Perspective, Here the focus is on the financial 

performance of the organization. Various ratios such as profit di- 
vided by sales would be examined. 

m. Manager's Perspective. This is a broad definition of productivity 
which includes quality, quantity, disruption, turnover, and absen- 
teeism (Katzell, Yankelovich, Fein, Ornati, & Nash, 1975). 

S. Industrial/Organizational Psychologist's Perspective. Here the 
concern is primarily the personnel subsystem of the organization, 
and the efficiency or effectiveness of that subsystem. 

Clfe-arly, these approaches are quite different. They measure different 
things, aM-d they are used for different purposes, They would also result in very 
different productivity measurement systems. 

Organizational Model Used 

A t: hird issue is that the measurement of productivity will be determined 
by the organizational model used. This issue comes from the literature on orga- 
nizational effectiveness. Campbell (1977) summarized the theoretical viewpoints 
on orgMUzsational effectiveness. While there are numerous models of organiza- 
tional effectiveness (Campbell, Bownas, Peterson, & Dunnette, 1974; Coulter 
1979; Engtey, 1977; Goodman & Penmngs, 1976; Mahoney & Frost, 1974; Mahoney & 
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Weitzel, 1969; Price, 1968; Steers, Porter, Mowday, & Stone, 1975), Campbell 
(1977) identified two general models: 



1. Goal Cantered, This model assumes that the way to assess or- 
ganizational effectiveness is to develop criterion measures assessing 
how well the organization's goals are being achieved. The goals to 
be assessed are referred to as operative goals. These goals are 
the ends sought through the actual operating policies of the organi- 
zation (Ferrow, 1961), as opposed to the officially stated goals of 
the organization (Keeley, 1978)* 

2. Natural Systems, This model assumes that the demands from the 
environment are so dynamic and complex that a finite number of 
goals cannot be defined. The theory holds that the organization 
should have the overall goal of maintaining its viability, without de- 
pleting its environment. This view focuses on the means of ob- 
taining organizational viability such as internal consistency, judicious 

istribution and use of resources, etc, The focus is on the people 
in the organization, not on the state of the organization's technology 
or its physical structure* 

These two global perspectives of organizations are quite different from 
one another, and imply quite different approaches to measuring organizational 
productivity. 

What To Include In The Measurement 

The next issue is what measures should be included in the measure of 
productivity, Clearly, the different perspectives such as the economist's and the 
accountant's have implications for what measures are included, as does the orga- 
nizational model used. There are, however, a variety of other possibilities* 
Campbell (1977), for example, listed 30 types of measures that have been used. 

L Overall Effectiveness 
2* Productivity 
3* Efficiency 

4. Profit 

5. Quality 

6. Accidents 
7* Growth 

8, Absenteeism 
9* Turnover 
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10* Job Satisfaction 

11, Motivation 

12, Morale 

13, Control 

14* Conflict/Cohesion 

15, Flexibility/ Adaptation 

16, Planning and Goal Setting 
17* Goal Consensus 

18. Internalization of Organizational Goals 

19* Role and Norm Congruence 

20- Managerial Interpersonal Skills 

21* Managerial Task Skills 

22* Information Management and Communications 

23* Readiness 

24. Utilization of Environment 

25* Evaluations by External Entities 

26, Stability 

27. Value of Human Resources 

28* Participation and Shared Influence 

29, Training and Development Emphasis 

30. Achievement Emphasis 

Seashore and Yuchtman (1967) reported on a factor analysis of or- 
ganizational productivity scores for insurance agencies. They identified ten fac- 
tors, many of which were quite different from those listed by Campbell. They 
included new member productivity, youthfulness of members, business mix, man- 
power growth, and market penetration, 

The variety of measures that could be included in a productivity measure- 
ment system clearly shows that no one set of measures constitutes productivity. 
The diversity of possible measures must be considered in the design of a pro- 
ductivity measurement system* 

Completeness Of The Measurement System 

In contrast to toe issue of what constitutes productivity, there is consider- 
able agreement that a productivity measurement system should include all impor- 
tant aspects of the organization's work, If the system is not complete, it could 
easily encourage neglect of those organizational objectives that are not included as 
part of the measurement system* In such a situation, the overall effectiveness 
of the organization would suffer. 
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Duerr (.1974) reported several instances of this problem* One case in* 
volved a manufacturing company* The plants in the company were compared, and 
incentives were given to supervisors and managers on the basis of an operating 
efficiency ratio. Certain aspects of productivity that were not included in this 
ratio were allowed to suffer so that the operating efficiency ratios would be fa- 
vorable. The result was that the company sustained significant economic loss 
because certain measures were neglected, 

One Air Force training manager described the affects of an incomplete 
productivity measurement system this way: 

We measure the things that are easy to measure but the greatest part of a 
mission is constituted by tilings that are not easy to measure ... [we] mea- 
sure those things that are easy to measure — and we measure them just 
great — but whether we achieve our mission will depend on these other 
objectives that we've avoided because we're not sure how to measure 
them. Until we come to grips with this sort of thing, we're not sure 
we're assessing the right thing* And if you don't assess the right thing 
you'll never know if you're productive (Tuttle, 1981, p. 24-25). 

Use Of An Overall Index Of Productivity 

Another broad issue for productivity measurement systems is the use of an 
overall index of productivity* We would argue that the use of a single index is 
very important because of its motivational value, A single index provides the 
members of the unit with a sense of improvement or decrement. The single in- 
dex would also seem beneficial for information purposes. A large number of 
pieces of information on organizational functioning can be very difficult to as- 
similate and use for making decisions. 

The problem of integrating numerous measures of productivity into a com- 
posite is similar to the problem of developing a composite measure of individual 
performance when several dimensions of performance are identified, Schmidt 
and Kaplan (1971) reviewed the controversy between composite and multiple cri- 
teria. Several points were made* two of which are relevant for productivity 
measurement: 

1. Composite criteria are useful for decision making, providing the 
weights applied to each of the criteria contained in the composite 
are subject to a verification procedure. 
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fac^that this may not be a valid assumption has been pointed out by Campbell 

...in the real world it is probably a mistake to think of effectiveness 

Sef J^^J^f ~ ° f ^ ^ to «• « at w^a" 
£1 * COntinuous Md linear Actions. For example higher 

and higher retention rates may be "irood" un tn * 71 » nigner 

"bad " ( p . 44). g p to a 90101 ^ men become 

n nm ^ heM POto SUffff6St combin iag or translating individual indices into a 
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Examples Of Productivity Measurement Ry^, 

Several authors have presented methods for measuring productivity These 

* *" conc ^ "sues that have been presented above, Hve 
of these methods will be presented here. 

One method is called Total Performance Measurement (Joint Financial 
Management Improvement Proffram iota* ^ • * . - Kjmm rinanciai 

dimtivitv th a ri, k rr f gram » 1976 ^ and is a technique for measuring pro- 
ductivity that has been used in a number of Government organizations This 
technique combines industrial engineering and behavioral sciencTSorfe ft 
measure various aspects of productivity. Objective productivity fSS^ES as 

comers of toe* III ^« tio ™ «• designed an^ admSstSS t 

^^^^TT^ aS itS The questionnaire data 

are related to the hard" productivity indices, and these data are then presented 
to management in a feedback session. The feedback session is leaned t 
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reveal the causes of good and poor organizational performance, and to form 
action plans, 

A second method was presented by Peeples (1978), who measured produc- 
tivity in 14 data processing centers. In this system, management weighted each 
aspect of productivity by assigning points for different levels of goal attainment. 
The more important goals could earn more total points than less important goals, 
Within each goal, more points could be earned as the organization approached the 
goal attainment level. The total number of points was the composite measure of 
productivity for each data processing center, The centers competed against one 
another for recognition in terms of total points earned. Significant increases in 
productivity and financial indices were reported, 

A third approach, by Felix and Riggs (1983), presented a productivity mea- 
surement system that depends on an "objectives matrix 51 to combine all productiv- 
ity criteria into a composite index. The objectives matrix matches different lev- 
els of performance on each productivity criterion with a performance level that 
ranges from one to ten. The highest goal level reasonably possible for each 
criterion would receive a performance score of 10, The lowest likely score for 
each criterion would receive a performance score of 0, and likewise for the 
points in between. Each criterion is weighted for importance, and the perfor- 
mance scores for each criterion are multiplied by the weights. The sum of the 
weighted performance scores is an overall index of organizational productivity 
for that unit. For an organization consisting of several units, the authors sug- 
gested weighting the overall productivity score for each unit by the number of 
people employed in each unit. The sum of these weighted overall productivity 
scores for the units is the aggregated productivity for the larger organization. 

A fourth method (Kim, 1980) is a technique for combining effectiveness 
and efficiency measures. This approach measures effectiveness by dividing each 
criterion score by the goal level for that criterion. Thus, each criterion has a 
percent effectiveness value. These percentages for each of the criteria are mul- 
tiplied by weights reflecting their importance; then they are summed to get a 
weighted composite effectiveness score, The weighted composite effectiveness 
score is divided by the number of criteria to get a weighted average composite 
effectiveness index. This final index is then divided by cost ratios to get ef- 
fectiveness-to-efficiency ratios. 

Finally, Tuttle and his associates (Tuttle, 1981; Tuttle & Weaver, 1986; 
Tuttle, Wilkinson, & Matthews, 1985) presented a detailed methodology for a 
partieipatively developed productivity measurement system. Their approach. 
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?MGEPlv£ ^ ^ eth0 *? < « r *?* Gyrating Efficiency and Effectiveness Measures 
(MQEEM), begins with the following steps: 

1. Management makes the decision to measure productivity, 

2. A measurement coordinator from within the organization is selected 

3. Researchers familiarize themselves with the organization. 

4. The boundaries of the organization are defined. 

5. An organizational diagram is constructed. 

•■wifJi! ^f* ° f mB Sy8tem requires managers and employees meet to 
identify the "key result areas" of the organization's performance, which corre- 
spond to the organization's objectives and the support activities for those objec- 
tives. Next, organizational members are asked for "indicators" or measures of 
those key result areas. Tuttle employs the nominal group technique to elicit 
ideas from supervisors and employees. Thus, the system reflects the ideas of 
those who will be using the system. Although he does not focus on it in detail 
Tuttle suggests that the indicators can be combined into a composite index using 
the matrix format described by Felix and Riggs (1983). 

Conclusions From The Literature 

In summary, there are many approaches, perspectives, and issues relevant 
to productivity measurement. It is tempting to ask, "What is the best definition 
and perspective to use in conceptualizing productivity?" However, we believe that 
this is the wrong question. Efficiency and effectiveness approaches both have 
their place, as do the different perspectives. How one resolves some of the 
other issues, such as how and what to measure, depends on the circumstances 
The better question is, "Under which circumstances is which approach most 
appropriate? For different purposes, very different approaches would be used. 

m. OUR APPROAC H TO ORGANIZATIONAL PRODUCTIVITY 

Following this line of reasoning, to establish our approach to organizational 
productivity, we first need to describe our purpose in measuring it In the sim- 
plest terms, our purpose in measuring productivity is to be able to increase it 
within a given organization or part of that organization. Our assumption is that 
the people m the organization have a great impact on the productivity of the orga- 
nization; that is, what they do and how they do it are most important. Although 
the technical subsystem is also important, our focus is not on that part of the 
system directly, but, rather on how the technical subsystem is used by the people 
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Therefore, to increase productivity, we need to increase the productivity of the 
people in the organization* 

The mechanism by which this increase would occur is primarily a motiva- 
tional one. That is, if motivated to do so, personnel would exert more effort 
and be more persistent in their efforts. They would work more efficiently in 
the sense toat their efforts would be more directly related to organizational ob- 
jectives* They \*w. ~ .mprove their work strategies and would use their own 
and others* time and efforts with less waste, This suggests that the perspective 
we will be taking is a combination of the managers perspective and the indus- 
trial/organizational psychologists perspective, 

Secondly, although we believe that both efficiency and effectiveness ap- 
proaches should be included in a productivity measurement system, we believe 
that the appropriate approach is first to consider productivity as effectiveness 
rather than efficiency. We take this position for three reasons, First, effec- 
tiveness—with its orientation toward goal attainment — is a broader definition of 
productivity, in that it results in a measurement system that expresses productiv- 
ity in terms of how good that productivity is. By contrast, an efficiency ap- 
p roach does not carry with it information as to what constitutes a good or bad 
level of efficiency. The second reason for our adopting the effectiveness ap- 
proach is that by taking this approach, we can more easily generate a measure- 
ment system th&t can combine all aspects of the organizations productivity into a 
single measure. The final reason is that the system we are proposing makes it 
possible to obtain an effectiveness measure and weight it by inputs to arrive at a 
system that combines the best aspects of both the effectiveness and the effi- 
ciency approaches, 

The organizational model we will be using is patterned more after the 
goal centered model than the natural systems model, The natural systems model 
emphasizes the interaction of the organization with its outside environment. This 
is clearly an important aspect of the long- range viability of an organization. 
However* issues such as how the organization will interact with the environment* 
what environmental changes will be forthcoming, and how to prepare for these 
changes are matters that are the responsibility of top management. Personnel 
from middle management down to incumbents do not typically get involved in such 
broad issues. Top management must make these decisions and then formulate 
plans and objectives for the rest of the organization to follow so that the organi- 
zation can successfully interact with its environment. 

Therefore, although it may be appropriate to consider the orgamzation 
based on the natural systems model at toe level of top management, the goal 
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cantered model is more appropriate if one is focused on the lower levels of the 
organization. This suggests that if the purpose of the productivity measurement 
is to increase the motivation of the members of the organization, especially from 
middle management down, the appropriate model to use is the goal centered 
model. 

Given our purpose of increasing productivity, it is critical that the mea- 
surement system be complete, so that increases in measured aspects of the work 
are not made at the expense of equally important but unmeasured aspects, Fi- 
nally, the individual measures should be combined into an overall measure of pro- 
ductivity for both motivational and informational purposes. This must be done in 
a manner that preserves the relative importance of the measures, and captures 
any non-linearity of the measures, 

Description Of The Productivity Measurement System 

The theoretical background for this approach to the measurement of orga- 
nizational productivity stems from the theory of organizational behavior presented 
by Naylor, Fritehard, and Ilgen (198G). In this theory, an individual's role is 
seen as a series of relationships, called cQntingencieg* These contingencies not 
only indicate what the important things are that the person must do in the job 
(called products), but also show the relationship between the amount of each of 
these activities and how that level of the product is evaluated. 

This approach to roles has the advantage of indicating more than the typi- 
cal information present in role specification. The typical information is limited 
to a listing of the important duties a person must perform on the job, In the 
Naylor, et al, approach this information is supplemented by the level of perfor- 
mance that is expected in each area, and how positively or negatively each level 
of performance is evaluated, 

In essence, we used the Naylor, et al, conceptual approach of using prod- 
ucts and contingencies, and extended its application from individuals to organi- 
zational units,, This application led to the development of a number of unique 
features for a productivity measurement system. We shall discuss these later in 
this paper, 

A second source for the development of our approach was based on the 
work of Tuttle (1981, Tuttle, et al M 1985), In this work, also supported by the 
Air Force Human Resources Laboratory, Tuttle developed an approach to mea- 
suring productivity that included methods of going from what we call products to 



so 



obtaining objective indicators of how well these products were being produced. 
He used a variety of group techniques, some of which we used also, 

Steps in the Development of the Productivity Measurement System, 

The technique to generate the productivity measurement system consists of 
four distinct steps: (a) identify salient products, (b) develop indicators of these 
products, (c) establish contingencies, and (d) put the system together. 

Step 1: Identify Products. 

Every organization has a set of activities that it is expected to perform. 
These activities result in a set of what Naylor f et al. (1980) called products. 
In using the term "product/" we mean more than merely a tangible thing that is 
produced. Products can be thought of as the set of objectives that the organiza- 
tion is expected to accomplish. The productivity of the organization is a func- 
tion of how effectively the organization generates these products. The first step 
in developing the productivity measurement system is to identify these products. 

To present the steps involved in developing the productivity measurement sys- 
tem, we shall use an extended hypothetical example that will make each step 
more concrete. For this example, we shall use a maintenance organization that 
diagnoses and repairs aircraft electronic communications equipment. The organi- 
zation's primary responsibility is to repair, as quickly and as accurately as pos- 
sible, the various items that are brought in when they malfunction. If a repaired 
item does not function properly when installed in the aircraft, it is returned to 
them for reaccomplishment of the repair. The unit is periodically inspected by a 
Quality Control function, which determines whether maintenance personnel are ac- 
curately following the procedures for repair detailed in available repair manuals, 
The maintenance unit also has responsibility for conducting on-the-job training, 
and a technician can repair a piece of equipment only if he/she has passed the 
training certification required for that piece of equipment. Thus, it is important 
that a sufficient number of people be qualified through training so that all the 
items can be repaired in a timely manner. 

To develop the system, the first step would be to meet with people from the 
organization to identify the salient products. Let us assume that the following 
products are identified: 



1. Quality of repair. 



2, Abili^ to meet demand for repairing items (i.e., the organization's 
ability to repair the needed equipment quickly). 



3, Ability to meet training needs (i.e., the degree to which the organi- 
zation meets its on-the-job training needs). 

In actual fact, there might well be more products in such an organization. 
However, since our Intent here is to explain the logic of the system, we shall 
use only these three so that the example remains simple enough for clear pre- 
sentation. 

Step 2: Develop Indicators, 

Once the products are determined, the next step is to develop indicators for 
each of these products. An indicator is a measure of how well the organization 
is generating the product in question. The indicators are determined from inter- 
action with the people in the organization, who are asked to think of those things 
which would show how well people in the organization are producing their prod- 
ucts. There may be only one indicator for a given product, or there may be 
more than one. Some indicators will already be available; some will have to be 
newly developed. After the indicators are discussed and refined, the products 
and indicators might look something like this: 

Product 1. Quality of repair. 

Indicator A: Return rate: percentage of items repaired that 

were returned for reaccomplishment of repair. 

Indicator B: Percentage of Quality Control inspections passed, 

Product 2, Ability to meet demand for repairing items. 

Indicator: Number of units repaired divided by total number 

of units brought in for repair. 

Product 3, Ability to meet training needs. 

Indicator: Number of people qualified to work on each type 

of item to be repaired, divided by number of 
people needed to be qualified. 
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As mentioned above, this would not be a complete list of products and in- 
dicators for such an organization, but it does serve to explain the concept of the 
productivity measurement system. 

Step 3. Establish Contingencies, 

Once the products and indicators have been identified and agreed upon, the 
next step is to establish the contingencies. A contingency is the relationship' be- 
tween the amount of the indicator and the effectiveness of that amount of the in- 
dicator. Figure 1 presents an example of a contingency. The top half of the 
figure shows the general form of a contingency. The horizontal axis represents 
the amount of the indicator, ranging from the worst possible level to the best 
possible level. For this example, we have chosen the first indicator: the per- 
centage of items returned for reaccomplishment of repair. Assume that the peo- 
ple in the organization have said that the best possible return rate is 2% because 
about 2% of the electronic components they use for repairs can work properly 
when installed and checked, but fail almost immediately when put into use. Let 
us also assume they said that the worst possible return rate would be 20%. 
Based on this information, values on the horizontal axis would fall between 2% 
and 20%. The vertical axis of the figure shows the effectiveness values of the 
various levels of the indicator. It ranges from +100 which is maximum effec- 
tiveness, to -100, minimum effectiveness. It also has a zero point which is de- 
fined as the expected or neutral level of effectiveness. That is, the zero point 
is neither positive nor negative. 

Once the best and worst possible levels of productivity have been estab- 
lished by the organizational personnel, the next task is to determine the zero 
point; that is, the indicator's expected level, the level that is neither especially 
good nor especially bad in terms of productivity. Once this is established, a 
point would be placed on the figure at the intersection of the zero point of the 
vertical axis and the level of neutral point on the horizontal axis. For example, 
if the neutral point were identified as a return rate of 10%, it would be indicated 
as shown in the bottom half of Figure 1, 

Next, the maximum and minimum effectiveness levels would be established. 
The first step is to list the maximums for each of the indicators. Assume the 
maximum indicator levels for the four indicators in our example were as fol- 
lows. 
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Indicator 



Maximum Possible Value 



1, Percent return rate 2% 

2, Percent quality inspections passed 100% 
3* Percent repair demand met 100% 
4, Percent qualified/needed 130% 

The group of incumbents and supervisors is then asked to rank order 
these maximums in terms of the contribution of each to the overall effectiveness 
of the unit. The one that the unit personnel believe to be the most important 
thing that the unit can do is given a rank of I, the second most important thing 
is given a rank of 2, etc* The group discusses this and consensus is reached. 
The maximum with the highest importance rank is then given an effectiveness 
value of +100, and the group is asked to rate toe other maximums relative to 
this. They are told to rate the other maximums as percentages of the +100 
maximum. For example, if the maximum of a given indicator was only half as 
important to the effectiveness of the unit as the most important maximum, they 
would give it a value of +50. An analogous process is then done for the mini- 
mum values of each indicator, except the most important (worst) minimum is not 
constrained to be a value of -100, it is given the value that the group feels is 
appropriate, 

- Once the zero points are identified and the effectiveness values of the 
maximums and minimums established, the remainder of the points in the line are 
developed by the group. Group discussion is continued until consensus is reached. 

Assume that the personnel in the organization said that return rate was an 
important aspect of their work, and that to be at the minimum (20% return 
rate) would correspond to an effectiveness of -80, and to be at the maximum 
(only 2% return rate) would be a +70* After the other points on the line were 
identified, this might result in a contingency similar to that shown in the bottom 
of the figure t which indicates that going above the neutral point results in in- 
creasing positive values* but such are not linear, In the example, once a return 
rate of 6% is reached, lower return rates do not represent as great an increase 
in effectiveness. Likewise, at the low end, once the return rate reaches 14%, 
they are doing very badly, and any rate below that is proportionally not as bad. 

After this process has been completed for each of the indicators, and once 
all have been scaled and reviewed for accuracy, the contingency set would be 
complete, * 
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tw* - SamPle COntin « enc y set is Presented in Figure 2. For each indicator 
there „ a contingency with its maximum, minimum, and expected or zero p^S 

rate, ■ tte «ne « that shown m the lower part of Figure 1. The second con- 
tmgeney „ for the percent of Quality Control inspections passed. Note that for 

Kecan°thlf!r y ^ ^ " that m% 0t *~ Sections be passe" 

Kecall that these inspections are not inspections of the final work but rather in- 
spections of the process the technician goes through in doing the repair ft?- L 
index of how well the person is following the manual in doing the repa rs It Z 
expected that all repairs will be done in accordance with the manual. Thus, tin's 
contingency shows that the expected level is doing all repairs (100%) in accor- 
dance with the manual. Anything less than this is below expectations and result 
m negate effectiveness. Note that in this particular case (since it s not ^ - 
ble to pass more than 100% of inspections), there are no positive values To- 
gether, these two contingencies cover Product 1, Quality of Repair. 

tor- f F f r* 2 ' AbUity t0 Meet Dema ° d f ° r Re Pairing Items, has only one indica- 
^k 01 ^ contin « enc y- The Indicator is the number of units re- 
paired divided by the total number brought in, expressed as a percentage This 
contingency is steep at the low and high end, and fairly flat in the middle sec- 

Product 3, Ability to Meet Training Needs, also has only one indicator: num- 
ber of people qualified (through training) to repair equipment, divided by the 
number needed, expressed as a percentage. For this indicator, it is possible to 
go above 100% qualified since, although the organization needs only 15 people to 
be quahf led to repair a given piece of equipment, it could actually have more 
tnan 15. However, the contingency becomes flat after 110%, indicating that hav- 
ing more than 110% is no more effective than having 110%. The idea is that 
once toere is a small excess over the maximum number needed, having additional 
trained personnel is not important. 

Two things are particularly noteworthy about these contingencies. The first 
is that the overall slope of the function expresses the relative importance of the 
indicator. For example, the overall slope for the first indicator (return rate) is 
steeper than that for the second indicator (percent of inspections passed). This 
reflects the fact that although it is important to pass inspections, which show 
that toe process of doing the repair was accurate, actually doing the repair so 
that the item functions properly is more important. Second, the contingencies can 
be non-hnear. As shall be discussed below, this is necessary to accurately re- 
flect toe realities of an organization's functioning. In many cases, the relation- 
snips that actually exist are simply not linear. 
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IFIGURE 2. EXA MPLE CONTINGENCY SET 
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important aspects of the organization and combine them into a sLte I Tn 

funot 0016 ff,° f ***** °' ^ meMUres - This amounts to Sa ^ SaU the 

sr.- 0ur r em d " sl t x 

tare of the contingencies. Aspects of the work that are very important 7et 
steeper contingencies than aspects that are less important. For eLmple ^ F f 

troJ'm* f ° r *** ° f Re P« r >- • range ""ff^e^ss 

IlO S J? 0, WherCaS Ability t0 Meet Training Needs ranges from S o 
+10. Tms indicates that Quality of Repair is more important iSn AbiSTto Meet 

StS; ^- variations in ^aHty of Repair have a greater 
Srenf asoITof th orgmz^ Thus, the relative importance of each dif- 
ferent aspect of the work is incorporated into the contingencies. 

into aPPr ° aCh C ° Uld be USed to Corporate differential importance 

mto the measurement system is to measure each aspect of the wort TcT^en 

div^^:^ 0 ^ -°- itS lmP ° rtanCe - < P ™^ each wo^ldt 
iirst dmded by its standard deviation to equalize the relative contribution before 

S^stinfr^ 080 - ThUS * f ° r CXamplC ' » <* Repair migM be tn 
sidered as being twice as important as Ability to Meet Training Needs To 7et 

by fanf -EST ^ ^ » ° f Rfi P air --sure^uld ^ mumplS 

by ^ and added to the training measure ™ c . «iuiu^iica 

* t * measure. We feel our approach is sunprinr 

Sere tT^J^^T * ^ ^ -««htinfmethod as" thS 
there is a linear relationship between amount of the measure and productivity- 
that is to improve a given amount at the low end of the measure 1 af ^ as 
improving that same amount at the high end. However, in the rea worlH Is 
very common for values in the middle range of an indicator to represent' large 
improvements in productivity, and values at the high end to represent a p^int o? 
diminishing returns. That is, once an organization gets to a fairly high l^ve of 

SST^i 011 T ° f W ° rk * " 18 fre ^ better to try to improve 

something else that they are not doing as well, rather than continue to mprove 
something that is already at a high level. improve 

™t* ^-Tu^u " *" " PaIr 8h0p Were grating with a very low return 
rate, « might be better to try to improve meeting its training needs rather San" 
attempting to further improve its return rate. Thus, even though return TaS 
overall is more important than training, if return rate is good, Im^nTa low 
degree of training readiness can become more important to tte overall ef- 
fectiveness of the org^tion. Another example of this non-linearity wouW be a 
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situation such as that depicted for training needs. Once the organization reaches 
a certain point, further increases are not more effective since all the people that 
are necessary are already trained. 

The simple weighting method ignores this non-linearity because no matter 
where the organization is on the measure, the value is always weighted by a con- 
stant amount, The contingencies in our system capture this non-linearity and thus 
provide a more accurate picture of the organization's functioning. 

Step 4, Put the System Together. 

Once the contingencies are completed and approved by management, the last 
step is to put the system together. This would be accomplished by first collect- 
ing the indicator data for a given period of time, If the time period selected 
were a month-long period, toe data for the four indicators would be collected at 
the end of the month, Then, based on the contingencies, effectiveness scores 
would be determined for each indicator by calculating the effectiveness for that 
level of the indicator. This is illustrated in Figure 3, For example, if the 
maintenance unit had a return rate of 6% in the month ©f March, examining the 
contingency indicates that such a return rate is associated with an effectiveness 
score of +60 (i.e., a value of 6% return rate on the horizontal axis is associated 
with an effectiveness value of +60 on the vertical axis), Continuing this process 
would give an effectiveness value for each indicator, as exemplified in Figure 3, 

Once the effectiveness values are determined, they can be summed to derive 
the overall effectiveness score for products with more than one indicator, as 
seen for the first product, The total effectiveness of the product Quality of Re- 
pair would be the sum of the two indicators comprising that product: +60 for 
Return Rate and -10 for Percent Quality Control Inspections Passed, for a total 
of +50, Next, overall productivity can be calculated by summing all of the ef- 
fectiveness scores. In the example, this Overall Effectiveness score is +20, 

These effectiveness scores have a distinct meaning, in that a score of 
zero means that the organization is meeting expectations; that is, their productiv- 
ity is neither particularly good nor bad. As the score becomes positive, they are 
exceeding expectations, The more positive the score, the more they are exceed- 
ing expectations. As the score becomes negative, they are below expectations. 
The closer they are to the maximum possible overall effectiveness score, the 
closer they are to their best possible productivity, 

Ibis ability to simply sum effectiveness scores is one of the major ad- 
vantages of the system. Because the contingencies reflect the relative importance 
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Figure 3 , Completed System. 



PRODUCTIVITY: MAINTENANCE UNIT 
DATE: March, 1987 



INDICATOR EFFECTIVENESS 

DATA; MARCH SCORE "~ 

I. Quality of Repair 

A, Return Rate $% _^ )0 

B. Percent Quality Control 95% _in 

Inspections Passed 

Total Effectiveness: Quality of Repair = +50 

II. Meeting Repair Demand 

A. Percent Demand Met 90% + jq 

III. Meeting Training Needs 

A. Percent Qualified/Needed 80% _4Q 



OVERALL EFFECTIVENESS = +20 
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and the non-linearity of the indicators, these factors are already incorporated in 
the system; thus, a simple summing does indeed reflect the overall effectiveness 
of the unit. As will be discussed later, this property also makes it possible for 
the system to be used to aggregate across individual units to determine the pro- 
ductivity of larger and larger units of the organization. 

Accuracy Of The System 

In order for this approach to be a good measure of organizational produc- 
tivity, it is obvious that it needs to be accurate. This means several things. It 
means that the listing of products and indicators must be complete. If there are 
important functions of the unit that are not included among the products, or if 
important indicators are omitted, the system can easily produce a situation where 
those t h ing s that are measured are attended to, and those that are not measured 
are somewhat ignored. This uneven attention to important functions can have 
very dysfunctional consequences for the organization, A second aspect of accu- 
racy deals with the degree to which toe system must accurately reflect what the 
unit should be doing. This means that the products, indicators, and contingencies 
must also be correct, 

Both completeness and accuracy are dealt with in the development of the 
system by having a clear process of approval of the system at higher levels of 
the organization, This approval process is made clear from the start. That is, 
at the beginning of the development of the system, all participants are told that 
incumbents and supervisors will develop the products and indicators, which will 
then be presented to higher management for approval, Once higher management 
has approved the products and indicators, the supervisory groups develop toe con- 
tingencies, which must also be formally approved by higher management. While 
this approval mechanism, and the multiple inputs that it provides, does not guar- 
antee completeness and accuracy of toe system, it provides a system of checks 
and balances so that the system will be as complete and accurate as possible, 

A final point about the quality of the system is that toe development of the 
system necessarily introduces subjectivity into the system. Subject! vity is present 
in the listing of toe products and indicators, and especially in toe ratings that are 
used in toe contingencies. Subjectivity is present, but this is not necessarily a 
problem, The elements of toe system— products, indicators, and contingencies- 
are in actuality statements of policy, As a whole, they say (a) what is impor- 
tant to toe functioning of toe unit, (b) toe level of output that is expected (toe 
zero point), (c) tow good other levels of ov!%ut are, and (d) toe relative impor- 
tance of different types of functions for toe unit. These determinations repre- 
sent policy, and policy is a subjective thing, ^ manager's primary responsibility 
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is to set policy, m the sense that he/she must determine the priorities for re- 
source allocation within his/her unit. What our system does is to reduce LS- 
guity « policy and priorities by formaily discussing them, '^^ZT^d 
~ng them to formal review and approval by the management ^ the £gaS- 

Priorities 

t, m thPT El % Z° 0ther UDlqUe featUrM 0f *» Productivity measurement sys- 
^m^to should be described; the first is the system's capability to generate Zt 

n^ T t he -5 Ste « °ff e " * Way *° d6Vel0p a clear set of Priorities for improving 
productivity. Recall from Figure 3 that for a given time period (e.g., . 

^ ^ Ste % P " SentS *f actual of each indicator achieved for that pertod. 

and the effectiveness levels of those amounts of the indicators. It would be a 
simple matter to look at the contingency for each indicator and caJcXe the 
2 ^SET tha l W ° Uld ° CCUr if *» «* w ~t UP one increment on each of 
JS^TE" ? 6 7* Peri0d ' F ° r eXampl6 ' if * e Unh had a Return Rate 
U 1% SLETp /? 18 f C f ted " FlgUrC 3 ' f0r them to «° to the next level up 
(a 4% Return Rate) m April would mean an increase in effectiveness from J> 
to 465 for a gam in effectiveness of + 5 units. This could be calculated for 
each mdicator. Once it was calculated, one could rank order the changed froZ 
highest to lowest. Such a listing for our example would look like that fn F £u« 

This information communicates exactly what should be changed to maxi- 
mize productivity. In the example it says that the best thing the umf can doTf o 

SET- ^ ° f ^ d6mand ' That is whereley should focus tL r 

efforts if they want to best increase their productivity. Once this is done or If 
increasing on this factor is not possible, the next best thing they could do is to 
improve training so that more people are qualified. Improving return rate and 
improving quality control inspections are the least important in inc^as^ 
Producuvity, with improving on quality control inspections beTng slight ™^ 
portant than improving return rate. 

t« m T H US, ^° + SyStem genCrate a S6t 0f P riori ties that unit personnel can use 
Lr/!„5°,- t0 fT" Pr0ductivity ' TUs wo «W aid in decisions about re= 
source allocation, and where to focus to identify barriers to productivity 
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Figure 4. Priorities for Increasing' Productivity. 



PRIORITIES FORi APRIL. 1987 

G^Al IN 

CHANGE EPFBCDTIVENESS 



Percent Demand Met from 90% to 100% +90 

Percent Qualified (Training) from 80% to 90% +20 

Percent Quality Control Inspections Passed 

from 95% to 100% + 10 

Return Rate from 6% to 4% +5 



Aggregation Across Units 



.Milt. A * S6COnd ""^ fCatUre of Privity measurement system is th- 



productivity measurement system for a given unit, or sav^r-aJ u^ts i t would hp 
even more valuable if one could aggregate measurement system fro m the seJ 
T£S? f " *• ^a/produe^^ 

tions, it would s 3: 1 tf ^it: iTti^jsri n, 8 -- 

combine *ose section measures into a s* LaLetr^e " fn 
most productivity measurement systems this is not possible- J^t th l ^ In 
jnents vary from unit to unit. An advance 0 f our a"^£ ""to * is"" 

«~ T egati ° n K ~ed on™on™ 

the scaling of the contingencies is d 0 n e with this aggregation in mind. 

If one were to simply add the overall effectiveness smcores of the differ 
ent sections to determine the productivity ofthe braneh, one woSd be eLenSlC 

Aitnougft tliis may indeed be the case, it is not safe to mmtaae it. It could easilv 
be that the work of one unit is more critical than that of * the others and Si 
to unit's effectiveness would contribute more to over* ef-fectiv^ness ttl^ 
of pothers. Another likely possibility is to a section ^ST^T^ 
to make more of a contribution to the ^ than a s^tion 4hTpelpS ? 

When iTZieTl T T^f " ^ * ffliriy ^ra*ightforward matter, 
wnen the system is developed for a single section and all 1 levels of suoervlsinn 

accurate for that unit. That is, because all personnel from ffincumbents to senior 

2T Scmc^^ ° n ^ C -~ S ' * «• —rate . refleSons of ~ 
icy Specifically, we assume that the valws for each conti«igency are accurate 
relative to the other values for that and that all 5e convene™ 

tta given section are accurate relative to Mother contingencies fof to section 
After the contingency set for each unit has teen developed, alLS to remans S b * 
done is to rescale the contingency sets for accuracy acr^a actions 

To explain this reseating across sections, let u s use an example of an 
pontes Branch in a Component Maintenance Division. Ass,™ to to br anch 

^tioT h ° ne sectioa » * e comim^to S-SS 

lection we have been using as the tuple; we wi ill call this unit 
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Communications. The branch also has two other sections, one doing maintenance 
on Inertial Navigation, and one maintaining Automatic Flight Control Instruments. 



To do the rescaling, we start with the fact the most important indicator 
for each section will have an effectiveness value of +100. This is tru -e simply 
because, in the development of the contingencies, the most important indicator is 
always defined to have an effectiveness value of +100. Thus, by definition, each 
of the sections in the same branch will have at least one indicator value of 
+100. In doing the actual rescaling, we take the indicator with the ■^+100 ef- 
fectiveness score for each section. This can be thought of as the mo&-t impor- 
tant indicator for each section. With the three sections in this bfancsh, there 
would be three such indicators (i.e., one for each section). 

The top indicators from the three sections would then be shown t~o branch 
management, as well as to managers from levels of supervision ambove the 
branch. The managers would then be asked as a group to rescale the tfc±iree lev- 
els. To do this, they first rank the three levels in terms of overall contribution 
to the branch. That is, they are asked which of the three outcomes th*.ey would 
most value for the overall effectiveness of the branch. In the example, they are 
given the three levels: 100% repair demand met in Communications, 100% - in Iner- 
tial Navigation, and 100% in Automatic Flight Control Instruments. They are then 
asked to indicate which of the three outcomes would make the greatest contribu- 
tion to the effectiveness of the Avionics Branch. They discuss this anil come to 
a consensus. It could be that they believe that since all three types otf compo- 
nents are crucial for an aircraft to be operational, they are all equally i important. 
In contrast, they may feel fhat the three are not equally important, T^his could 
come about for a variety of reasons. For example, one section migttt repair 
more items and have a correspondingly larger number of personnel. TThus, that 
section makes a larger contribution to the overall functioning of the br«anch than 
the other sections. Differential importance could also occur because ttSiere is a 
sufficient backlog of repaired components in supply for two of the sections, but 
not for the third. Thus, repair of the non-backlogged components is naoore cru- 
cial for meeting mission requirements than repair of components for whnich there 
is already a sufficient backlog. 

Assume that in the discussions, unit supervisors and managers d«ecide that 
meeting 100% repair demand in Communications is the most important thing for 
the branch, meeting 100% repair demand in Automatic Flight Control In»struments 
is next, and meeting 100% repair demand in Inertial Navigation is next. 

Once this ranking is completed, the indicator ranked Mghesst in ef- 
fectiveness is given a value of +100, and managers are then are aska-sd to rate 
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tne remaining indicators relative to this one. In doing this, they c told to think 
in terms of percentages; that is, they should ask themselves whether the second 
most important thing is 95% as important as the most important^ 90%, etc As- 
sume that the second most important maximum is given a value of 90 and the 
third a value of 75. This means that the group is saying that meeting 100% re- 
pair demand in Automatic Flight Control Instruments is almost as important as 
meeting 100% repair demand in Communications, but not quite ass important In 
fact, they are saying that it is 10% less important. Meeting 1Q0SS& repair demand 
m Inertia! Navigation is somewhat less important, and in fact is ■ only 3/4 as im- 
portant as meeting demand in Communications. 

Once the relative ratings of the top indicators for the sections are agreed 
upon, the next step is to rescale the individual contingencies for e=ach of the sec- 
tions. This is done by reducing the effectiveness score of e a c«i level of each 
indicator in a given section by the percentage its own maximum indicator was re- 
duced in the rescaling. For example, in Automatic Flight Control Instruments the 
original +100 maximum was reduced to +90 in the reseating process This rep 
resents a decrease of 10%. In essence, it is saying that to be essomoarable with 
the other sections in the branch, the effectiveness of this level mof the indicator 
must be reduced by 10%, since it is not quite as important to the- branch as the 
maximum of the Communications section. Since the effectiveness value of the 
maximum was reduced by 10%, in order to retain accuracy it is necessary that 
the effectiveness levels of each of the indicators for that section, also be 

reduced by 10%. This means that if the original positive values of one of the 
indicators for that section were +10, +20, +40, and +75. the veaiues after the 
rescaling process would be +9, +18, +36, and +67,5. This process of reduction 
by 10% is continued for each contingency in that section. 

A similar process is then done for the Inertial Navigation ^section Here 
the maximum was reduced by 25%, from +100 to +75. Thus, each positive ef- 
fectiveness score for each contingency in that section must be reduced by 25%. 

Finally, the effectiveness values for the Communications -^section do not 
change. Since the original maximum of +100 was unchanged in tthe process of 
rescaling across sections, the effectiveness values for the contingencies in Com- 
munications are not recalculated. 

An analogous rescaliug process is then done for the negative^ effectiveness 
values of the indicators. The most negative level of the indicators is listed for 
each of the branch's three sections. These three levels are then ranked as to 
which constitutes the poorest level of effectiveness. Finally, ju&t as with the 
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po^isitive values, the nipUva valutas of each level of the indicators are each ad- 
ju^sted by the percentage that the ^original minimum indicator level was reduced* 

This reselling process hate the effect of adjusting the effectiveness scores 
of ' the different sections in the Bforanch for any differences in importance of the 
difirferent sections* Om it if fi— nished, the overall effectiveness values from the 
diSfferent sections are calculated _ Just as before. The only difference is that the 
efcf ectiveness scores for the dif f^arent levels of the indicators = have been rescaled 
btesed on the aggr^gatta process^ - The overall effectiveness score from each of 
th^3 sections can now simply be summed to produce overall effectiveness of the 
enrtire branch. For example* if -the monthly overall effectiveness for Communis 
captions was +250* for Internal ^Navigation was +150, and for Automatic Flight 
Ctezitrol Instruments was ~fhe total branch effectiveness would be +600, 

Tfcziis value can be interpreted jus=rt like overall effectiveness for a single section. 
If it is 0, the branch overall L- s meeting expectations. If it is above 0, the 
brsanch is exceeding expectations ^snd the higher it is above 0, the greater they are 
exwceeding expectations . 

This approach to aggregatL*on can be extended to larger and larger units of 
th^ organization, so that, if desir-ed, a single index of toe productivity of the en- 
tire organization can be developed. For example, one could aggregate branches 
ijriao a division ind^x. Assume th^at in the Component Maintenance Division that we 
ha^^e been discussing, there is r^ot only an Avionics branch like that described 
abeove* but there is also a second branch called Propulsion, This branch has two 
sanctions* jet Engines and Test CJnit, jet Engines repairs and rebuilds engines, 
Tost Unit runs the repaired/reKwilt engines in a test facility to evaluate the 
fUKtctioning of the engine, 

To aggregate this second ^iranch up to the level of the division, the first 
st«ps are to develop the products, indicators, and contingencies for the two sec- 
ti&MLS in the Propulsion Branch* * Once this is done and the system for each sec- 
tio^n is approved up the chain of command t the next step is to aggregate the two 
sanctions into the measure for t^ae Propulsion Branch, and to aggregate the two 
brsmches into a measure for the GComponent Maintenance Division. 

To achieve th%se two levels of aggregation (section to branch and branch 
to division) is fairly straightfot — ward. In essence, we do the same thing that 
was done for tb^ Avionics BranoXfa* Instead of rescaling the three maximums 
frcom the three sections in AviOMucs, we rescale the five maximums from the 
/i»e sections in the Component ftSaintenance Division at the same time* Put an- 
other way, if oue ^ants to fl^^gregate up to the level of the Division, the 
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aggrt^%tion to t=3ie branch and to the division is done at the same time, and by the 
same process. 
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Specifically, we would take the maximum indicator value from the five 
sectioiMj and liast them, just as we did when aggregating the three sections of 
Avionics to the branch index. This would result in a list of the five levels of 
the indicators tfaat received the +100 effectiveness value. For the three sections 
in Avionics, thi=s would be meeting 100% of repair demand in the three sections 
of Comjiiuiicatieons, Inertial Navigation, and Automatic Flight Control Instruments, 
Added to these three would be the indicator levels in the two sections in the 
Propulsion branseh that had received the +100 effectiveness scores. Assume that 
for the Jet Et^mes section the indicator level with the maximum effectiveness 
value w»s having 6 or more jet engines repaired, inspected, and ready for 

installation, As==sume that for the Test Unit section, the indicator level with the 
+100 effectiveotess score was having 0% engines that had been passed by the sec- 
tion returned as malfunctioning. 

r 

Ttee fiv^e maximum indicator values would be ranked and rated just as in 
the exmit of rising only the three maximums from the Avionics branch, As^ 
sume that the raftings came out as follows, 



Section Maximum 



Rating 



C^Qjth*ations 100% Repair demand met 100 

Jet Begins 6 or more engines ready for installation 98 

Test tMt 0% engines returned as malfunctioning 95 

Auto Flight Control 100% Rep air demand met 90 

Inertial Navigation 100% Repair demand met 75 

Ui other wvords, the indicators from the Propulsion Branch were seen by 
the group as slightly less important to the functioning of the division than the 
most important Exiaximum from the Avionics Branch, but more important than the 
other itx^imums frofti the Avionics Branch, Once these values have been deter- 
mined, tie final step is to recalculate the effectiveness values of the indicators, 
an wn§ dona in ^*he previous example. Effectiveness scores for the Communica- 
tions section wowjld remain unchanged since the original maximum with its value 
of +100 is still *100 after resealing, The effectiveness values for the indicators 
in the otfcer four- sections change, Each positive value in Jet Engines is reduced 
3%, S% hi Test I3nit, 10% in Auto Flight Control, and 23% in Inertial Navigation, 
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As before, an analogous process is done with the indicator values with 
negative effectiveness scores, The five negatives are ranked and rated by the 
group and then the negative effectiveness scores are rescaled by the percentage 
that the maximum for that section was reduced, 

Once the rescaling of the indicator values is completed, overall ef- 
fectiveness for each section can be calculated as usual, using the rescaled effec- 
tiveness scores, Branch overall effectiveness is simply the sum of the section 
overall effectiveness scores, and division overall effectiveness is simply the sum 
of the two branch overall effectiveness scores. By going through this process of 
rescaling, the sections in each branch and the branches in the division are made 
comparable to each other. This simple summing of overall effectiveness scores 
preserves the relative importance of the different sections and branches. 

This same logic of rescaling to make the different units comparable with 
each other can be continued to larger and larger units. In theory, it is possible 
to use this approach to develop a single index of productivity for the entire Air 
Force. 

One potential problem that could come up in this rescaling process is that 
the shape of the original contingency could change, Recall that rescaling is done 
on the positive effectiveness values, then repeated for the negative effectiveness 
values. In other words, the rescaled effectiveness values for the best possible 
levels of the indicators are determined, then the process is repeated for rescal- 
ing the effectiveness values for the worst possible levels of the indicators, 

This two step process could have the effect of changing the shape of the 
contingency that was originally developed by the unit and its supervision, For ex- 
ample, suppose a given section developed a contingency that was linear. That is, 
the contingency was a straight line from the worst level of the indicator (with 
e,g. a -75 effectiveness value) to the best level of the indicator (with a +75 ef- 
fectiveness value). After rescaling, the maximum could stay at +75, while the 
minimum was rescaled to ^50, This would mean that the rescaled contingency 
was no longer linear in shape. It would be steeper from the zero point to +75 
than it would be from the zero point to ^50, 

This in and of itself is not a problem. The new non-linear contingency 
would be the most accurate reflection of the contribution of amounts of that indi- 
cator to overall organizational effectiveness, and the new contingency would be 
used in calculating effectiveness values for unit feedback reports, The potential 
problem is that the unit personnel developed a contingency that has now been 
changed by the aggregation process, This could lead to a lack of acceptance of 
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the new contingency by unit personnel,' Tbous, it is important to explain to unit 

changs the contingencies. In addition, if contingencies are changed the reasons 
for these changes should be expired to uoit personnel by th^'t Superiors 
who were present at the meetings that did th^e resealing. supervisors 

One final point should be made about this resealing process to enable M 
gregation to larger units to be done. The description of ftfrescaHnT process" 

T^^ST*"**- bM ^ * -ally W ite ^ ToSy 

requires that the appropriate supervisors anrf managers be brought together for 

^St^ s for the sectionsTbe % 

gregated While tins can be a difficult set of judgments for them to make it 

^kes only a short amount of time. This is .specially true since at 4^ in 
me process these personnel h a v e be«,hvolv«d in the development of the System 

for some time and should be quite familiar with the issues. Th e TLtTZ 

wcud expect such meetings to take would be two hours. Once the judges are 

m^de it is a simple matter to recalculate the effectiveness scores. Once tis L 
done, the resealing is finished. 

TV. APPLICATION Of THE SYSTEM 

t,,«ii v Alth ° Ugh We believed VProalto measuring productivity to be concept 

^nWT T lf " W0 ^ d ^ be ~^ - - ^ 

ar^o^ if f d tMt ° f *» Sy8tem w - undertaken 

as pait of an AFHRL-funded effort coaducte*J by the authors at an operational 
Air Force base in the Southwestern Wad States. In the overall produS 

rSaEirr 1 °H ^ tateP ~ 8f «*• setting/and'in^ves 

as techmques for enhancing organisitiona! productivity. The productivitv mea 
surement system described here served as the, basis ?or the feedback ^1 ™ 
ELS *e crtterioaby ^hich the three interventio^w^e 

T^v , t> "rterventions and their results, are presented in another AFHRL 
Technical Paper (Fritchard, Jones, Roth, Stuafaing, & Bkeberg, 1987). Here we 
are focusing on the productivity measurement system itself. 

The organizational units involved in the afield test consisted of one mainte- 

wTs the^L^ SeC V* ta *— " The maintenance uSt 

was the^Commumcations and Navigatioa siotioMi (Comm/Nav) in the Component 

cn^n ^ l° Ur SWtl0M ««*^ce Management, which toS 

oomprised the Materiel Storage and Distribution Branch (MS&d) of the S^y 

^ o r n n ' Were ReceiVi ^' Stor ^ . Pickup and Delivery, an7^ 

jAggrsgatlon techniques which preserve t h | functional form 0 f the or1a< na 7 , ont< 
are theoretically possible but have not b e e n fully txpl ored as part of thZ .*«- \ contingencies 
work on this topic is planned at AFHRL. P 5 P * rt 0f th1s « ffort * Additional 
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The Comm/Nav section was similar to toe example of the electronics 
maintenance unit used above in describing the development of the system, Its 
function was to repair a variety of electronic equipment used for aircraft com- 
munication and navigation. The MS&D branch was essentially the base ware- 
house* Property was delivered to the warehouse and checked in by the Receiving 
section; Storage and Issue shelved toe property and retrieved it as it was ordered 
by units on toe base; the Delivery section had the responsibility for delivering the 
property to units on base that had ordered it; and Inspection's responsibility was 
to make sure toe property was in good condition, and ensure that regulations 
were being followed concerning packaging, storage, and identification of property. 

System Development 

Once toe five units for study were selected, the first step in the develop- 
ment of the system was to, meet with supervision and management of the twits 
and explain the purpose of toe project and what toe research team would be do- 
ing. The development of toe productivity measurement system was also ex- 
plained, and questions were answered, After this, meetings were held jointly 
with incumbents and supervisors to actually develop the system, Most of the 
system development was done through group meetings, These meetings were held 
on visits to the base made approximately every two weeks over a period of six 
to nine months, 

Once toe missions of toe units were examined, toe first step was to iden- 
tify toe products of each of the units. This task proved to be a fairly time- 
consuming one. There was considerable debate on exactly what the important 
products of the units were, We continued these meetings on products until con- 
sensus was reached. 

The same process was used in the identification of the indicators. The 
meetings were held over a considerable period of time since there was a great 
deal of discussion and initial disagreement as to what indicators would be appro- 
priate. During this process, it became clear that examining the actual data from 
various proposed indicators would be of value. This process was very instruc- 
tive to the unit personnel, They realized that some existing measures were in 
fact measuring quite different things than they had thought, and they found mea- 
sures that they did not know were available, It also became clear that a number 
of new measures would have to be collected to get a set of indicators that would 
adequately represent all the important functions of toe unit, As with the devel- 
opment of the products, toe meetings were held approximately every two weeks 
over a period of three to six months. 



One issue that surfaced with the MS&D branch Involved an indicator that 
was a very important one but could not be directly attributed to a single section. 
The indicator involved the number of "delinquent documents," warehouse property 
documents that, due to some error, were lost in the system. When this occurs 
these documents must be searched out and the problem rectified. The problem 
with an individual document could be caused by any of the sections in the ware- 
house; this complicated the determination of which section should be responsible 
At the same time, delinquent documents were very important to the ooeration of 
ihe warenouse, and needed to be included in the system. 

The resolution of the issue was to have an indicator for the number of 
delinquent documents, but make it an indicator that applied only to branch produc- 
tivity. This meant that the effectiveness score for the branch would be influ- 
enced by how well or how poorly the four sections did on this indicator; how- 
ever, the effectiveness of the individual sections would not be affected This 
approach proved successful, in that the number of delinquent documents was re- 
duced substantially, and no individual section felt that it was being held account- 
able for another section's errors. 

Once the products and indicators had been finalized by group consensus 
the next step was to get approval of the products and indicators from higher^ 
level management. This approval was obtained using the entire chain of com- 
mand from the units up to and including the Deputy Commander for Maintenance 
for Comm/Nav, and the Deputy Commander for Resource Management for MS&D 
These personnel were again briefed on the project, and given written versions of 
the proposed products and indicators. They were given time to study these doc- 
uments, after which a meeting was scheduled for formal review. In this meet- 
ing, the products and indicators were presented, discussed in d^ail, and, after 
some revisions, approved. 

The next step was the development of the contingencies for each indicator 
Meetings were held with ir .mbents and supervisors of each unit. First, the 
maximum and minimum ind' tor levels were established; then, the zero points 
were generated. These dec ions also took considerable time, resulted in con- 
siderable initial disagreement, and were arrived at over several meetings Once 
consensus was established, the effectiveness scaling of the indicator values 
within each unit was started. 

This scaling was started by listing the maximum possible value for each 
sf the indicators for their unit. The group was then asked to rank order these 
naximums in terms of contribution to the overall effectiveness of the unit. The 
rroup finally reached consensus. The maximum value assigned the highest 
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ranking was then given an effectiveness value of +100, and the group was then 
asked to rate the effectiveness values of the other maximums relative to this. 
They were told to rate the other maximums as percentages of the +100 maxi- 
mum* For example, if the maximum of a given indicator was only half as im- 
portant to the effectiveness of the unit as the most important maximum, they 
were told to give it a value of +50, This process was also done as a group by 
unit supervisors and incumbents, and continued until consensus was reached. An 
analogous process was done for the minimum values of each indicator, except 
that the least effective minimum was not constrained to a value of -100, but was 
given the value that the group felt was appropriate. 



Aggregation Across Sections 



Once the contingencies were developed in each section of MS&D, the seal- 
ing required to aggregate across the four, sections to the branch level needed to 
be done* As described above* this involved first identifying the indicator value 
scaled as +100 in effectiveness in each section; i.e M the maximum indicator value 
for the most important or top indicator for each section. These are listed be- 
low. 



Receiving: Get Priority 2 property (the highest pri- 

ority property other than extremely rare 
emergency aircraft parts) to Pickup and 
Delivery in an average of 10 minutes. 



Storage and Issue: Get Priority 2 property to Pickup and 

Delivery in an average of 10 minutes. 

Pickup and Delivery: Get Priority 2 property to customers on 

base in an average of 15 minutes, 

Inspection: Have 100% of the aircraft parts inspected 

by 4:00 PM each day* 



These four indicator values were next ranked by supervision and manage- 
ment as to their importance to the overall functioning of the branch. The first 
three were ranked as equally important* and toe fourth was ranked below these. 
The three number one ranks were then given an effectiveness value of +100; that 
is, their effectiveness values remained unchanged. The supervisors were next 
asked to rate the last indicator value. They felt its importance in terms of ef- 
fectiveness was very close to the first three* and gave it a value of +98, 
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hv ^"f^ 6 . ffeCtlVeneSS scores for e ach contingency were then reduced 

by the « the original maximum had been reduced in the reseaiing/ Since toe 
maximum for Inspection had been reduced from + 100 to + 98, eaehlLt^^ 
of each contingency for this section was reduced by 2% of its o"riSX %2S 
value For example, an original scaling of + 50 would be rescaled ^% Be 
cause toe posit.ve effectiveness values remained unchanged for toe other sections 
the contingencies for these sections could be used intact! sections, 

™, H I*?* + f 0CMS ' tte four secti °os were scaled so that effectiveness values 
could be direetiy compared across sections. In addition, it was now possibfe to 
sum toe overall effectiveness of toe four sections to get an overall ef fitness 
for toe branch that reflected relative importance of toe four sectos The 

^ Sch sum" ZTJ? ,0 V* miDOr adjUStmWt ^ what would have been 
toe branch sums had the resoaling not been done. However, this says only that 
m tots particular branch, toe section maximums were regarded as very simHar to 
toeir contribution to branch productivity. This will not "always be Te eas^d 
the rescahng could result in major changes in values for toe other units. 

Manageme nt Approval Qf System 

fr„ m C ° nSenSUS WM reached on mm contingencies, approval was obtained 

from higher management, using toe same process as described above for toe 
products and mdicators. Many questions were raised by higher management 
umt supervision m toe meetings on contingencies, as well as in toe meeting" on 
products and indicators. Higher management asked for clarification of many 
points and wanted to hear toe units' defense of toeir system. While toe majority 
toe If ' indlCat0rs ' «* contingencies were left as originally developed by 

toe units, there were some changes made as a result of toe approval meetings 
The process was a positive one. The resulting discussion clarified toe position 
of higher management to toe units' supervisors, and the positions of toe supervi- 
sors were made clearer to higher management. In all cases, toe resulting 
changes represented compromises that seemed to satisfy both groups. 

n p . £ e f mp ^ ° f a C0 «Pleted system of products and indicators is presented 
n Figure 5. This system is toe one that was developed for toe Comm/Nav 
namtenance umt. It served as toe basis for toe example used earlier in this pa- 
►er, however, as can be seen, toe actual system is more complex than toe sim- 
plified example. The final system had 3 products and a total of 13 indicators 
rhe products were Equipment Repair, which includes both quality and quantity 
measures; Triumng; and Other Duties, which includes several miscellaneous ac- 
uities of varying importance. Figures 6 and 7 present examples of actual con- 
ngencies that were developed for the Comm/Nav unit. Figure 6 shows toe 
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Figure 5, Comm/Nav Products and Indicators. 



Product 1. Equipment Repair 

Bounces* Percentage of repaired equipment that did not function 
*" immediately after installation, 

Percent QA (Quality Assurance) inspections passed, 

AWM: Number of units awaiting maintenance, 

AWP: Number of units awaiting parts* 

Demand Met: Percentage of equipment brought in for repairs that 
was actually repaired, 

Product 2. Training 

STS Tasks Completed: Mean number of standard (more basic) 
training tasks completed for personnel in training. 

Percent Qua! Tasks Completed, Comm: Mean percent of advanced 
training tasks completed for personnel repairing 
communications equipment* 

Percent Qua! Tasks Completed, Nav: Mean percent of advanced 
training tasks completed for personnel repairing navigation 
equipment* 

Scheduled Training Tasks Overdue: Total number nontechnical 

(e.g* military) training requirements not met on time tor all 
shop personnel. 

Products, Other Duties 

Mobility Equipment: Number of pieces of equipment used for 

mobility exercises that were not calibrated by the shop on 
schedule, 

PMEL Overdue: Number of pieces of shop calibration and tejt 

equipment that were not calibrated by the shop on schedule. 

Percent 349 Errors: Percent of errors on a major manpower 
documentation form. 

Missed Appointments: Number of formal on-base appointments 
missed, 
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contogancy for "Bounces" (i.e., the percentage of equipment where the renair 
needed to be reaccomplished). This is an indicator foT Equipment Rep * r t s 
a very important one, as shown by its range in effectiveness from I to -I 
^^ WS *• for Mobility Equipment Overdue Vibrato 

This acuity is less important than Bounces, as shown by its more shaHow 
slope wrd, it. range from +25 to -50. The complete set of conUng^es for °U 
five of the units are presented in Appendix A. ungencies tor all 



Feedback 



f«J^f u I r S t0 m tte SyStem to » ether to Produce productivity 
feedback for each of the units. In doing this, we developed examples of what 
*e basic system would look like and asked for management's mouihts on how 
best to present the material for maximum clarify. We also proposed f smne oS 
information that they might find useful, and asked them for ttS suggest on 
things to be included. This was discussed, and after several reviskms TfinS 
version of the productivity feedback report was developed. An exa^e of "I 
m*my report for the Comm/Nav shop is presented in Figure 8. Samples of tte 
feedback reports for the four sections of MS&D are presented in Appendix B 

^ Pag ° ° f ^ repOTt P rovldes the basic productivity data It 

shows the products and indicators, the indicator data for that month, Ld Se eft 
fewness score associated with that level of each indicator. The lower portion 

overall T ^ t0tal effectiv ^» tor each of the products and f inaHy te 

overall effectiveness for the stop. The second page of the report adds infor! 
ma^on to the basic data. The top half of the report shows the change in pr^- 
ductmty from the previous month to the current month. The indicaSr data and 
effectiveness scores for both the previous month and the current month "e 

Jj™U I ^ WM fe y shop personnel to aid them in di- 

agnosing areas where they were increasing or decreasing in productivity. 

producm^^ ^ ? ^ ^ 18 toformatio11 ^ priorities for increasing 
S t FROM f °T iS * C ° lumn lab6led FR0M . TO, and 

Tn?TO colu^r " r OUnt ° f Mcat0r for * e «™« ^nth. 
unif 1 * ^ aa ° Unt ° f tte indiCat0r ^ resents an increase of one 

ma " u ^ b r«T n !H J" *! GAIN column indicates the gain in effectiveness 
tta, would be achieved by such an increase. For example, for Demand Met, if 
the shop went from their March level of 91,7% to 9S.2% in April, effectiveness 

^tfT^ £ 37 , P ° int3 - »— *■ of the GAIN c^rf^S^ 
tiie next month the shop would increase their productivity most effectively by fo^ 
cusing on Quality Assurance Inspections, and the number of units Awaiting Parts 
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Figure 8. Sample Feedback Report. 



PRODUCTIVITY REPORT 
COMM/NAV SHOP 
CR5 MAINTENANCE 



INDICATOR AND EFFECTIVENESS DATA FOR MARCH 1986 



nnacATOR 

EQUIPMENT REPAIR 
BOUNCES 

% OA INSPECTIONS PASSED 

AWM 

AWP 

DEMAND MET 



INDICATOR 
DATA 



3.1 
90.9 
13,5 
39,6 
91.7 



TRAINING 

STS TASKS COMPLETED 
% QUAE TASKS COMPLETED- COM 
% DUAL TASKS COMPLETED: NAV 
SCHED TRAINING TASKS OVERDUE 



8 
69.5 
56.8 

0 



EFFECTIVENESS 
SCORE 



76 
30 
80 
29 
63 



35 
72 
68 
10 



OTHER DUTIES 

MOBILITY EQUIPMENT OVERDUE 

PMEL OVERDUE 

% 349 ERRORS 

MISSED APPOINTMENTS 



0 
0 
I 
0 



25 
25 
40 
10 



TOTALS 



EFFECTIVENESS 
SCORE 



EQUIPMENT REPAIR 
TRAINING 
OTHER DUTIES 

OVERALL EFFECTIVENESS 



278 
185 
100 

563 
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Figure 8 . (Concluded) 



EPFECTIVBNBS3 CHANGE FROM FEBRUARY TO MARCH 



BOUNCES 

% QA INSPECTIONS 
AWM 
AWP 

DEMAND MET 



INDICATOR 

DATA 
FEBRUARY 

2.8 
91.7 
15.58 
40,6 
91.5 



STS TASKS COMPLETED 9 
%QUAL TASK3-GOMM 68 6 
%QUAL TASKS-NAV 59.5 
SCHEDULED TRAINING 
TASKS OVERDUE 0 

MOBILITY EQUIPMENT 

OVERDUE o 

PMEL OVERDUE 0 

% 349 ERRORS 2 

MISSED APPOINTMENTS 0 



EPF. 
SCORE 

81 
34 
72 
27 
59 

52 
72 
71 

10 



25 
25 
27 
10 



INDICATOR 
DATA 
MARCH 



0 
0 
1 
0 



CHANGE TOTALS 



EQUIPMENT REPAIR 
TRAINING 
OTHER DUTIES 
OVERALL EFFECTIVENESS 



EFF. CHANGE 
SCORE IN EFF. 



3.1 


76 


-5 


90,9 


30 


-4 


13.5 


80 


8 


39.6 


29 


2 


91.7 


63 


4 


8 


35 


-17 


69.5 


72 


0 


56.8 


68 


-3 


0 


10 


0 



25 
25 
40 
10 



0 
0 
13 
0 

5 

-20 
13 
-2 



POTENTT A T„ EFFECTIVENESS GAINS FOR NEXT MONTH 



BOUNCES 
QA INSPECTIONS 
AWM 
AWP 

DEMAND MET 

STS TASKS COMPLETED 
%QUAL TASKS COMP: COMM 
%QUAL TASKS COMP: NAV 
SCHED TRNING TSKS OVERDUE 

MOBILITY EQ OVERDUE 
PMEL OVERDUE 
349 ERRORS 

MISSED APPOINTMENTS 



FROM 


TO 


GAIN 


3.1 


0,4 


17 


90.9 


100 


45 


13.5 


0 


15 


39.6 


22.6 


48 


91,7 


95.2 


37 


8 


9 


17 


69.5 


76 


0 


56.8 


62.8 


4 


0 


0 


0 


0 


0 


0 


0 


0 


0 


1 


0 


15 


0 


0 


0 
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(AWP). These show potential gains in effectiveness of 43 and 48 respectively. 
It would not be useful to devote attention to training in Comm Qualification 
Tasks, trying to further decrease Overdue Scheduled Training Tasks, or any of 
the other indicators that have a gain value of zero or near zero. This informa- 
tion can therefore serve as a basis for determining priorities for the next month* 
The unit should focus on those areas where the maximum gain in effectiveness 
could be produced. 

The calculation of the GAIN amount is based on the amount of increase in 
effectiveness that would occur with an increase of "one unit" of the indicator* 
The size of a one unit increase was determined from toe indicator values used 
in the contingencies* If toe indicator values in a contingency were 2%, 4%, 6%, 
8%, etc. the size of a one unit increase for that indicator was 2%. If the indi- 
cator values were 50, 60, 70, 80, etc., toe size of a one unit increase was 10. 
The contingencies were originally developed so that toe number of increments for 
the different contingencies was as equal as possible so that a "one unit" incre- 
ment was roughly comparable across toe different contingencies. Once the size 
of the "one unit" increase was determined for each contingency, the TO figure 
was calculated by adding toe one unit increase to the actual value of the indicator 
for the preceding month* If the last month's indicator level was 83.6 and toe 
size of one unit was 10, toe FROM value would be 83,6, the TO value would be 
93,6, and toe GAIN value would be determined by what toe contingency Indicated 
as the gain in effectiveness if toe unit went from 83.6 to 93.6 on that indicator. 

There was one special circumstance that had to be dealt with using this 
approach, It was possible for the TO value to be higher than toe maximum 
value of toe indicator. This occurred when the unit was already high on that in- 
dicator and increasing "one unit" would put them over toe maximum. It also oc- 
curred occasionally if the unit was already over the maximum on that indicator* 
This was dealt with by using the maximum possible effectiveness value for toe 
indicator as the upper limit in effectiveness. In other words, if the effective- 
ness value for being at the maximum of toe indicator was +75, this was the 
maximum effectiveness score that could be gained from that indicator. If toe 
unit was near the maximum already with, for example, a past month's indicator 
level which yielded an effectiveness score of +73, toe most they could improve 
would be to the value of the ceiling, +75, for a maximum gain of only +2. 

The feedback report for MS&D was similar to this Comm/Nav report ex- 
cept that the report for each of the four sections included information on how 
the entire branch did for the month, including indicator and effectiveness data on 
toe branch-level indicator discussed previously. In addition, toe MS&D report 



42 51 



ERIC 



added a feature that allowed for a direct comparison of the four units to each 
other. 

One feature of the system is that it allows one to directly compare the 
productivity of very different units, This feature was very important to the su- 
pervisors and managers of the MS&D branch since it allowed them to compare 
the four sections of the branch. In order to make this comparison, we first 
determined the maximum possible overall effectiveness for each section. This 
was done by determining the effectiveness value for the maximum possible value 
of each indicator, and summing these effectiveness values. The resultant score 
represented the effectiveness value that would occur if the unit was doing as 
well as it was possible to do on every aspect of their work; in other words, 
their maximum possible effectiveness. Recall that these maximums were devel- 
oped by consensus among the supervisors of the units, and discussed and approved 
by management. Thus, they should represent realistic maximums, and the effec- 
tiveness scores represent the value of the maximum contribution each of the units 
could make to the organization. 

Once the maximum possible effectiveness was calculated, the actual 
monthly overall effectiveness score for each section was expressed as a percent- 
age of maximum possible effectiveness. This percentage of maximum effective- 
ness was the measure by which each unit was compared to the other, These 
data were included in the monthly feedback report for each section of MS&D, 

The feedback report was generated each month for each of the five units 
for the 15 months of formal intervention by the research team. It was pre- 
sented within three workdays after the end of the month, and a copy was given 
to each individual in the chain of command, from the section supervisors to the 
Deputy Commander, A copy was also posted in the working area of each section 
so that incumbents could review it, In addition, graphs were posted in the work 
area and updated each month: one for overall effectiveness, and one for each 
indicator, These graphs allowed unit personnel to see changes in effectiveness 
over time, As one might imagine, both the feedback report and the graphs gen- 
erated considerable interest when they were posted each month. 

Once the feedback reports were circulated, a meeting was held with sec- 
tion supervisors and a representative from upper management to review the feed- 
back report for toe month. Areas of improvement were noted, and areas of de- 
crease discussed* Reasons for the improvements or decreases were considered, 
and any longer-range trends were noted. This meeting also served as a basis 
for planning priorities for toe next month, and for making changes to improve 
productivity* 
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This feedback of the information from the productivity measurement sys- 
tem was provided for five months in each of the five units. After this period, 
goal setting was added to the feedback. After five months of feedback plus goal 
setting* incentives were added in the form of time off for superior productivity. 
The details of these interventions are presented in a separate paper (Prit chard, 
et al,* 1987). 

Results 

The results of this application of the productivity measurement system are 
presented in three sections: (1) results during development of toe system, (2) 
effects on productivity* and (3) effects after the departure of the research team, 

1. Results during development 

One of the most interesting results that occurred during the development 
of the system was the change in the attitudes of the unit personnel. When we 
first started working with them, their attitudes toward the project were mixed* 
Although some unit personnel were positive, others were more skeptical. By the 
time system development was completed, however, almost all unit personnel had 
positive attitudes toward the effort. They were solidly behind the system, felt 
positive toward the researchers, and were quite disappointed when the start of 
feedback had to be delayed so that enough time had passed to establish a base- 
line. 

Development of the system also resulted in a conscious examination of unit 
objectives* the development of possible measures of these objectives, and an eval- 
uation of productivity expectations and limits* This process led unit supervisors 
to see numerous places where improvements could be made in the operation of 
the units. Naturally enough, they began to implement these changes. This cre- 
ated a real dilemma for the researchers. Although it was certainly worthwhile 
for the units to improve their effectiveness as a result of the development of the 
system* this improvement was occurring prior to the establishment of our base- 
line. If because of this, the baseline period showed higher effectiveness than it 
would have otherwise, this would decrease the size of any effect due to the pro- 
ductivity feedback* There was little the researchers could do about this 
dilemma. The units felt strongly that such changes should be made* and made 
theriu They felt these changes were increasing their effectiveness, and this in- 
deed seemed to be the case* based on what little data were available at the time. 
Interviews with supervisors indicated they believed that a substantial portion of 
this improvement was due to the process of developing the productivity 
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measurement system. This suggests that the improvements in productivity that 
were evidenced in me interventions were, in fact, underestimates of the overall 
impact of the development and introduction of the feedback from the productivity 
measurement system. 

Another issue concerning the development of the system had to do with the 
difficulty of me task for me personnel involved. We had expected the greatest 
difficulty to be in the development of the contingencies. We were quite mis- 
taken. The development of the indicators was the most difficult. 

One problem with the development of the indicators was that unit person- 
nel tended to be very accepting of existing measures. This was not surprising, 
since they had not been trained to critically evaluate possible measures and were 
somewhat resistant to change. For example, one measure considered by 
Comm/Nav later turned out to contain elements due to supply and flightline 
maintenance. Only about 10% of the variance in the indicator was due to factors 
under the control of Comm/Nav. They had been using this as an indicator of 
Comm/Nav effectiveness, but when its actual content was identified, they dropped 
it as a measure. 

Another problem in developing indicators was that unit personnel did not 
always see the implications of certain measures. For example, the Receiving 
section of MS&D must input information about each piece of incoming property 
into the computer. If a mistake is made in this process, a "reject" is later 
printed out by the computer. Unit personnel must then identify the cause of each 
of these rejects and correct the data. The indicator mat was first proposed was 
the percent of rejects that were cleared from the reject list each day. At first, 
mis seemed to be a quite reasonable index. However, after studying these lists, 
it became clear that some rejects were quite easy to clear, whereas others were 
extremely difficult. Thus, if percent cleared was used, the better the unit did on 
clearing rejects, the more they would be dealing with only the most difficult and 
time-consuming rejects. In the long run, this would automatically lead to a 
poorer percent cleared, and thus be a poor indicator. 

In contrast, the development of the contingencies went very smoothly. As 
mentioned above, we had expected contingency generation to be quite difficult for 
the personnel involved. Contingencies are conceptually complex, and we felt the 
effectiveness scaling would be an especially difficult task for mem to do. What 
in fact happened was that after the contingencies were explained, unit personnel 
had little difficulty in developing them. They seemed to grasp very quickly what 
contingencies were and how they would be useful. They told us that contingen- 
cies captured the way they mought about what they did, and in a more clear and 
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comprehensive way than they had ever done before, They also said that the con- 
tingencies facilitated communication by providing a meaningful frame of refer- 
ence for discussions of policy. Personnel became very involved in contingency 
development, and frequently made revisions in the developing contingencies be- 
tween our visits to the base* 

Reliability and validity were also assessed during the development of the 
system, Reliability was assessed by interjudge agreement on the contingencies. 
The Comm/Nav shop had two shifts in operation. Personnel from both shifts 
were involved with the development of products and indicators. To assess the 
reliability of the contingencies, we developed two independent sets of contingen- 
cies, one set with each shift. This produced two effectiveness scores for each 
value of each indicator: one set from the day shift and one set from the night 
shift, Correlations were calculated between the two sets of values for each 
contingency. For example* if a given contingency had eight levels of the indica- 
tor* there would be eight effectiveness scores developed by the day shift and an 
independent set of eight effectiveness scores developed by the night shift. The 
two sets of eight scores were then correlated, producing one correlation for each 
contingency. These correlations ranged from a low of .86 to a high of ,99, with 
an average of ,95, Thus, the reliability of the contingencies as measured by in- 
terjudge agreement was quite high. 

The validity of the system was evaluated using five different productivity 
scenarios of hypothetical indicator data developed for Comm/Nav, This was done 
by selecting a reasonable value for each indicator in such a way that the differ- 
ent scenarios varied as to their overall effectiveness. Although the overall ef- 
fectiveness of each of the five scenarios varied* the differences were not so 
large as to be completely obvious. Also* the changes in indicator values varied* 
but not always in the same direction. That is, although the overall effectiveness 
may have increased for a given scenario, some indicator values went down, while 
others went up. Six Comm/Nav supervisors were then given the indicator data 
on the five scenarios and asked to rank the scenarios as to their overall effec- 
tiveness. If toe system is accurately reflecting relative importance, having su- 
pervisors rate the scenarios without knowing the scenario overall effectiveness 
scores should produce ratings which are highly correlated with overall effective- 
ness as calculated by the system. 

These ratings were done approximately two months after the development 
of the system had been completed* but before any productivity feedback had 
started. Results showed a correlation of LO between each supervisor's rankings 
and the overall effectiveness score calculated by the system. This constitutes 
additional evidence for the validity of the system. 
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2. Effects on productivity 



Once the system was developed and a baseline established, the system was 
used to generate feedback. Next, goal setting was added to feedback, and fi- 
nally, mcentives were added. The overall effects across the five units are 
shown m Figure 9. As the figure indicates, overall effectiveness increased sub- 
stantially over the baseline. The average increase over baseline was 50% for 
feedback, 75% for goal setting, and 76% for incentives. Figure 10 presents sim- 

? *?,L 0o,nm/Nav d**- The data show average increases of 30% for 
feedback, 65% for goal setting, and 68% for incentives. Figure 11 presents the 
data tor MssD alone. Average percent increases were 54%, 77%, and 79%. 

These data indicate a major increase in productivity. The effects were 
extremely large. In addition, the MS&D effects were consistent across sections 
fcach of the four sections showed a consistent pattern of increase for feedback 
and even greater increases for goal setting and incentives. It would be worth- 
while to present a sample of these changes in terms of indicator data to give a 
sense of their magnitude. The most important indicator for Comm/Nav, percent 
of repair demand met, had a mean of 88.5 during baseline. This mean became 
90.5 during feedback, 93.9 during goal setting, and 92.6 during incentives. The 
most important indicators for MS&D were times to process Priority 2 property 
Ihe mean time to move a piece of property was 92.6 minutes during baseline" 
This figure was cut to 25.3 minutes during feedback, and to 19.6 and 19.8 min- 
utes for goal setting and incentives, respectively. 

These results were then examined in light of the level of manpower in the 
five experimental units during the period to determine if the changes in produc- 
tivity that occurred during the treatments could possibly be due to changes in unit 
manpower. For Comm/Nav, the data examined were the total number of person- 
nel in the shop each month. Our analysis indicated that mean number of per- 
sonnel^ during baseline was 30.9. This figure increased slightly during feedback 
to 33.0, was 32.8 during goal setting, and dropped back to 31.0 during incentives 
Since manpower levels during the period of highest gain in productivity were es- 
sentially equal to the level during baseline, we concluded that the increases in 
sroductivity were not caused by increased manpower. In MS&D, the data re- 
newed were total number of personnel, and the number of hours of overtime 
egged per month. Unlike Comm/Nav, MS&D routinely had considerable overtime 
The mean level of manpower for MS&D was 51.8 for baseline, 53.7 for feed- 
Jack, 48.4 for goal setting, and 49.2 for incentives. Thus, overall manpower de- 
ceased over the period of the treatments. In addition, number of hours of 
overtime decreased from a mean of 1,343 hours during baseline to 892 during 
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. "* gMl Wld 416 *"** taenUves. Thus, by the end of 

¥Lr^ ^° Vert, r W8S ' eSS *" «— of w ^t i, was during oesefine 
These data indicate that the productivity gains that occurred were male wi* ^ 
increase in manpower in Comm/Nav, and a decrease in manpower u/mS&IX 

Results were also examined in light of productivity data from several s.r 
t,ons S ,mHar to the experimental sections. These sections then, in essence" 
served as a control group, since they had no intervention from the research t^™' 
Control data for Comm/Nav consisted „, 10 measures of e?2 
%nr? UmtS , ln C^ 1 ™** Repair Squadron. The data wte coital 

tte co^r 6 ^ 'V? * C ° mP0Site meaSUre to Wress overall prolctiv ^ oi 
tte control groups. This composite measure was the sum of the tan measure/ 
m mean value across the 10 measures during baseline was 317; « d" 3 * 

retpecUvX. ' ,0 ^ 365 *"« g0i " » d "as! 

These results show that the productivity of the control sections decreased 
somewhat dunng the Comm/Nav feedback period, and increased *e™tt„ TUs 
would suggest mat the productivity increase during feedback in Comm/Nav was 
Crieff 2 ™ I .optional changes since other squadron units actually de- 
creased during this period. Furthermore, the increases during the Comm/Nav 
goal setting and incentives periods were not present across all^ita ThTe 

of T S fen Vere , b ? Ugh ' ab °"'P rim ™» * large increases in product vij on two 
ot the ten control measures. 

Th M , ^ r0dUetivi ^ ^ ata on four c ontrol measures were examined for MS&D 
Jf MUm aSUr f ° VeraU functi ^ of the Supply Squadron (exclusive 

llT^ ' ""k ^ 0th6r ^ ° UtSide ° f Supp1 ^ The ™» ^thase four mel! 
suras dur«g basehne was 516; it was 512 for feedback, 511 for goal seSng, 
and 518 for incentives. Thus, there were essentially no changes in productivhy 
for the sections outside of MS&D. H y 

tw Tak °^ t ? ge ^ er » control d «t« indicate that the effects on productivity 
that occurred m the experimental units eannot be explained by wider oTg^ 
tional changes in productivity, organiza 

^ WerC . alS0 c ° llected on objective reactions to the system. All incum- 
bents and supervisors (M7 ) were administered a survey after several i 

pects P o e rZ%Lt ^ f8edbaCk Sy8tem - ThCy Wer ° flsked to differed As- 
pects of the feedback system using 5-point Likert scales with response formats 

ra^ng from Strongly Agree to Strongly Disagree. Responses toTufvey" 

were umformly positive. Table i presents the items and the percentage or 
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Table 1. Subjective Evaluation of the System 



ITEM 



PERCENT. 
AGREE OR 
STRONGLY 
AGREE 



PERCENT 
DISAGREE OR 
STRONGLY 
DISAGREE 



1. The feedback system tells me how 

good a job I am doing, 64 

2. The feedback system tells me how 

good a job the section is doing, 87 

3. The feedback system helps me see 

the section's priorities. 77 

4. The feedback system helps the 

section be more productive. 61 

5. A system like this would help other 

Air Force bases be more productive, 62 

6. The feedback system is clear and 
understandable. 58 

7. It was worth the effort to develop 

the feedback system. 64 

8. It was worth the effort to keep the 
feedback system in operation. 62 

9. The information about section 
performance that goes into the feedback 
system is accurate. 52 

10. The feedback system gives a good 
measure of productivity. 64 

11. Overall, I like the feedback system. 62 

12. I would prefer not to have this 
feedback system at the next organization 

I work in, 13 

13. The feedback system is a better 
way of measuring productivity than what 

the section used to have. 75 



MEANS = 64.7 



4 

1 

6 

4 

4 

7 

10 

10 

13 

13 

9 

54 
0 

7.2 
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i ungiy oisagreed. After reverse scoring the negatively worded item f#m a. 

.Tree^s 7« 6 cLT:r n<Hl,g PerCen,ag8 W "° DiM * r « d « Strongly^ 
ver^ poXe * reSP ° nSe *° *« sys,em "sed ft was 

3 - Effects after the depart,.,-.. of fl, a rBg>arnh 

~.r* 8 e rr e vr re s:aur~ k oi C e a 7.n sr. 

treatment was over, our on-base responsibilities officially ended Wv, 
were on base for a variety of purposes after ttis time ie u^s'had nc com' 

researcner, efT ■£? M"" Un " 5 eIeCMd t0 continue *• after the 

*e system ' r<!S0UrCeS 10 d ° " ^ »*» that the units ' P u7e« on 

At the end of the formal incentives treatment, units were asked if tW 
wished to modify the system. If they did want to, we agreed^to asdst iflt f 
cam. necessary. Comm/Nav and two sections of ^^VS 
tern ?h 6e ? Se f' tt \ chan * es * ere to eliminate some indicators f rom 1n7s^ 

f£" J T 1 Wer ° r ° m0Ved WCre ^ with very flat contingencies 

(indicating they were not very important), those pertaining to activiti^S S 

™ ^roTtSevT 8 t0 T f ° rm ' ° r *~ *- "~ tO bTund^ 
good control that they were no longer important to measure, 

ily As S^JT^^ ^. SyStem is toat » can "commodate changes read- 
ily. As changes occur in policies, procedures, or resources, changes will need to 
be made in the system. This can be done by eliminate indices rede^nin^ 
them or altering the scaling of contingencies. Thus, the system can ^ Z 
altered to changing conditions. However, it must be undersSd ^afaSr Leh 

"fir^ vene8s scores are ™^™ZrZ£Vt 

will T ^P 1 *' lf indicators are dropped, the same actual productivity 

loit toae^eiete?^ 0 ?^ 1 ^i***™*** — • «• affectiveness'points" e 
lost to the deletes mdicators. This makes the interpretation of effects over 
time dsfftcult until a new "baseline" is established. A new baselinf Ts estlb 
hshed by taking the newly revised system of indicators and 
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calculating its overall effectiveness for several months prior to the revision, 
For example, if some indicators are dropped, it is a straightforward matter to 
go back to the indicator data from prior months and calculate what overall ef- 
fectiveness would have been if those indicators had not been included. This then 
becomes the new baseline, and the effectiveness scores after the revision are 
completely mterpretable. If indicators are added, it is a matter of recalculating 
what the overall effectiveness for prior months would have been had these indi- 
cators been included* This is a simple matter, provided historical data are 
available. 

One feature of the system that needed to be explored was whether the 
units would be able to use the system after the researchers left* As part of 
that process, the units needed to be able to make changes in the system, since 
the need for changes would undoubtedly occur in the future* Thus, we were 
particularly interested in their ability to do this, In making these changes, it be- 
came clear that the management of the units understood the system fairly well, 
and with help from the research team were able to make the changes. With this 
help, they were able to eliminate the indicators they wished, adjust the calcula- 
tions of effectiveness to take the removal of the indicators into account, and re- 
calculate past effectiveness to establish a new baseline. Our assessment is that 
Comm/Nav could now make such changes completely on their own, and M5&D 
could with minimal help. 

Before we left the base, toe units wanted us to train them to use the 
system, so that they could continue using it after we left, This proved to be a 
fairly simple task in Comm/Nav* By the end of the incentives treatment, they 
had already taken over the collection of all the data that went into the system. 
They had only to be trained on using the microcomputer programs that were used 
to calculate effectiveness scores and generate the feedback reports. This was 
done readily, and other than an occasional question, they have been operating the 
system completely on their own for several months, 

The training in MS&D was more involved, Although their intent to take 
over the system had been frequently expressed for some time, the actual com- 
mitment of personnel was not made until the end of the incentive treatment* 
Thus, the training could not be done gradually over many months, as would have 
been optimal. In addition, the task of preparing the feedback reports in MS&D 
was more difficult than in Comm/Nav* The MS&D reports required the entering 
of data stowing toe amount of time it has taken to move property in toe ware- 
house. Someone must take several hours each month to enter these data and run 
toe program that calculates the mean times for toe indicator data* During the 
interventions, this was done by toe research team, After our departure, it had to 
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FIG. 12 PRODUCTIVITY 
AFTER THE DEPARTURE 
OF THE RESEARCH TEAM 
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major index of the system f s completeness was that f across the five units, not one 
indicator needed to be added during the 20 months of experience with the system. 

The system is also valid in terms of its ability to accurately reflect how 
well the unit is functioning* For the system to be valid, the products must be 
correct, the indicators must be good indices of the products, and the contingencies 
must be accurate. The structure of the system allows for high validity because 
the relative importance of different activities is maintained, and non-linearities 
are preserved* The iterative process of development of products, indicators, and 
contingencies promoted the type of repeated review that maximizes accuracy* 
Higher management's approval ensured that the system was an accurate reflection 
of organizational policy. Finally, the reliability and validity data obtained, were 
highly supportive of the validity of toe system. 

Another apparent reason for the success of the system was that it allowed 
for direct comparison across units. That is, the system allows units that have 
totally different functions to be directly compared as to their level of productiv- 
ity* This feature was utilized considerably by the management of MSM), Hav- 
ing each unit's overall effectiveness expressed in terms of percent of maximum 
possible effectiveness, it was easy for management to assess how well the units 
were doing relative to each other; for example, to make decisions about resource 
allocation across units. 

Another important feature of tins system is its flexibility . The system is 
flexible in that it can accommodate both effectiveness and efficiency approaches 
to productivity, The effectiveness approach is the one used here, and the effec- 
tiveness scores are an expression of output relative to expectations. However, 
the system can also accommodate an efficiency approach by incorporating ef- 
ficiency into the indicators. For example, a measure of Oomm/Nav monthly re- 
pair demand met divided by the manhours available that month would be an effi- 
ciency measure of labor productivity, Thus, efficiency can be included in the in- 
dicators. 

In addition, efficiency can be included by taking the overall effectiveness 
measure and dividing by total inputs, This becomes a measure which combiner 
efficiency with effectiveness, and may be a very valuable approach in many situ- 
ations. For example, overall effectiveness could be divided by the total costs 
over which &e unit has control (costs of manpower, supplies, etc) to obtain a 
measure of cost effectiveness, Unit personnel would then be expected to maxi- 
mize cost effectiveness by increasing effectiveness, decreasing costs, or both. 
Yet another approach would be to divide overall effectiveness by the total costs 
of operating the unit whether controllable by the unit or not. Such a cost 
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effectiveness measure would be a major component in a management information 
system. Finally, multiple measures could be used to get a more complete picture 
of the functioning of a unit. 

The system is also flexible in terms of its ability to accommodate changes 
within an organization. There will be changes over time in any organization's 
objectives or procedures. These could be brought about by changes in policy, 
changes in technology, or refinements in operations. The productivity measure- 
ment system must be able to accommodate these changes when they occur. 
This system can accommodate changes through modification of any of its compo- 
nents. Products, indicators, or contingencies can be changed. For example, if a 
unit decided to change from an emphasis on quantity to an emphasis on quality, 
the contingencies for the indicators of quality could be made steeper, and the 
slopes of the quantity indicators less steep. Or, if expectations change, changes 
can be made in the zero points of some of the contingencies, or the entire con- 
tingency can be rescaled. 

An important advantage of the system is that it can be applied to any level 
of the organization, and can be aggregated to larger and larger parts of the orga- 
nization, as we did in MS&D. In principle, this aggregation could be continued 
until the entire organization is included, and its effectiveness could be expressed 
in a single number. 

In our field test, we chose to develop the system at the lowest level in the 
organization. We did so because we wanted to affect the motivation of unit per- 
sonnel and, from our previous work on feedback, had concluded that the maxi- 
mum impact on motivation comes about when feedback is directly relevant to the 
specific tasks ttiat are being done. It is also possible, however, to develop the 
system at much higher levels of the organization. One could develop products, 
indicators, and contingencies at the squadron level, for example. Such an ap- 
proach would not have the motivational features of being able to give specific 
feedback down to the lower levels, but it would serve as a basis for under- 
standing and monitoring the produetiviQr of the larger unit. 

The ideal strategy would be to start at the lowest level and build the sys- 
tem for each section, then aggregate up to the level of the higher unit (e.g. the 
squadron or the wing). This would have the advantage of providing motivational 
impact to the individual sections and branches, and also provide information on 
the functioning of the entire squadron or wing. 
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Motivational Value 



A second factor that seamed to make the system effective was its strong 
motivational value* First of all f the system resulted in a clarification of rales. 
The process of developing* refining, and getting management approval for the 
products, indicators, and contingencies forced role clarification* When the pro- 
cess was finished, the units had a clear picture of what their objectives were, 
what they should be focusing on to achieve these objectives, what was expected 
of them in each area, and what good and bad productivity were in each area. 
Thus, they had a far better understanding of their roles than they had before the 
development process started* This role clarification process likely had positive 
motivational effects in and of itself* 

The use of the system also dramatically increased the amount and quality 
of feedback unit personnel received, With the system, they received more accu- 
rate feedback, and more positive feedback; when the feedback was negative, it 
was provided in a useful, non-personal form, The positive feedback seemed 
particularly important, Personnel indicated that the system represented one of 
the few times they had been told they were doing a good job. The productivity 
measurement system provides a great deal of positive feedback* When the unit 
is doing well or when it is improving, the reports show this, This positive 
feedback is a very important feature of the system* 

Another motivational feature of the system is that personnel could see the 
results of their eff orts , Most jobs are structured such that doing a better job 
does not show up in any measurable way* This can become very demotivating* 
The frequent feedback provided by the system seemed to improve the connection 
between individual effort and unit productivity, Personnel could see the effects 
of their efforts to improve productivity, 

The system also allowed for competitio n ac ross units , Using this system, 
units can be compared in terms of percr ,\ -:>f am'^um effectiveness, and can 
thus compete on this basis. In MS&D, mm^ y&i- */■. clearly present between 
sections, was friendly in nature, and s^ax^-a to ; ^ c positive effect on produc- 
tivity. 

One of the most important features of the system was that it allowed 
personnel to focus on the same objectives. Before implementation of the system, 
different supervisors and different levels of management focused their efforts 
on different tilings. Furthermore, what was high priority changed frequently. 
One week a great deal of effort would be put on one thing, and the next week 



something different would be top priority. This is a problem common to most 
organizations, It is brought about by people having different ideas about the 
umts* priorities. It is especially problematic when there is no agreement on pri- 
orities by different levels of management. Constantly changing priorities create 
serious problems for unit personnel. 

The process of developing the productivity measurement system seemed to 
reduce this serious problem dramatically, In essence, all levels of management 
had agreed on what was and was not important. All levels had an opportunity to 
see the perspective of the others, all had agreed on what was important, and a 
concerted effort could then be made to accomplish the organization's objectives. 

Organizational Accountability 



Mother factor in the system's effectiveness was that the system made 
units accountable for their productivity. Units can be held more accountable when 
their productivity is measured and reported on a ragular basis. Concrete per- 
formance data are hard to ignore, The fact that the data exist and will continue 
to be provided generates a source of motivation for unit personnel to want to 
look good; also, they know that they will have to answer for it if they do 
poorly. However, whenever productivity data are presented objectively about 
problem areas, as they are in this system, there is much less of a tendency to 
make excuses, and more of a desire to try to find positive solutions. 

Another aspect of accountability is that the system allows for a way of 
assessing the effectiveness of supervisors and managers. Their major responsi- 
bility is to effectively use the resources under their control to maximize the 
achievement of the organization's objectives. Thus, one very useful index of 
their effectiveness is the effectiveness of their unit. 

Feedback 

Another apparent factor in the effectiveness of the system wap that it - 
provided considerable information for identification and diagnosis of problems . 
The feedback reports showed if productivity had started to slip in a given area. 
This allowed the unit to deal with the problem before it became serious. Prior 
to receiving feedback, a problem could become much more pronounced before it 
came to the attention of the unit, In addition, the feedback reports aided in diag- 
nosing the causes of problems, Finally, as mentioned above, the feedback al- 
lowed the unit to know when a problem was fixed, 
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The system also gave information on appropriate priorities . The section 
of the feedback report that indicates where the maximum increases in ef- 
fectiveness would result served as a guide for setting priorities and allocating 
resources for the upcoming month, 

VI, CONCLUSIONS 



A number of conclusions can be drawn from this research. First we will 
discuss specific research conclusions concerning the capabilities of the system. 
We will then discuss issues concerning the productivity measurement system de- 
velopment process* Finally, we will review other potential applications of this 
technique* 

Specific Research Conclusions on System Capabilities 

The system evaluated in the present effort appears to be a very effective 
way of measuring productivity, Its implementation is quite feasible* unit person- 
nel were cooperative in developing the system, and it showed good psychometric 
properties, In addition, the system development process itself seemed to have a 
positive effect on unit functioning even before feedback was instituted* 

The feedback that resulted from the productivity measurement system had 
a very strong effect on productivity, An average gain in productivity of 50% oc- 
curred across the five units during feedback, and gains of 75% and 76% occurred 
when using feedback with goal setting and incentive interventions, respectively. 
The positive effects of the system have lasted over time* Specifically, the pos- 
itive productivity results continued for the 15 months that the research team was 
present on base, and have continued for at least 5 months after that. 

The system allowed for aggregation across units so that an integrated 
system could be developed across the four sections making up a branch, This 
process is actually quite simple once the basic system is developed in each sec- 
tion. The application of this aggregation to much larger and more complex orga- 
nizational units seems quite feasible. 

The five units using the system evaluated it very positively. Expressed 
attitudes toward the system ranged from positive to very positive. The members 
of the units were quite proud of the system and their productivity improvements* 
For example, unit members frequently showed the system to people visiting the 
unit* In addition, after the experimental period, all of the units continued using 
the system on their own, and management wanted to expand it to other units as 
well. 
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The development of the system had positive effects for units that were 
quite different from one another in terms of the nature of the work, The ac- 
tivities were quite different between Comm/Nav and MS&D, as well as among 
the four sections of MS&D. There were also great differences in the type or 
organizational structure, and the work flow. The units varied considerably in 
size, and the personnel varied considerably as to their academic and technical 
training background. They also differed as to their initial levels of 
performance. Yet, with all these differences, the system was developed and 
worked extremely well in each unit. This adds considerable support to the gen- 
eralizabili^ of the approach. 

The system also offers a. number of other benefits. It allows for the di- 
rect comparison of the productivity of different units to each other. It can be 
used with both effectiveness and efficiency approaches to productivity. In addi- 
tion, it can be applied to any level of the organization, allows for competition 
between different units, helps identify the priorities for increasing productivity, 
and serves as an excellent basis for evaluating any changes made in the orga- 
nization. 

Conclusi ons About the Productivity System Development Process 

In doing this research, a number of co- lusions were drawn concerning the 
process used for developing productivity measurement and enhancement programs 
in Air Force environments. 

One is the importance of having the personnel who are going to be using 
the system be heavily involved in its development. The perception of unit per- 
sonnel was that some previous programs imposed from above had not been ef- 
fective because these programs were not designed with an appreciation for their 
unique needs and environment. It is much more effective to have heavy involve- 
ment from the personnel that are going to be using the system, so that the final 
product will fit their needs and they do not feel that it is yet one more project 
imposed from above. 

It also seems more effective to develop such programs from the bottom of 
the organization up. The lower levels of supervision know the most about the 
functioning of the unit, and what are the real critical issues. In addition, these 
are the people that will make the system work. It is important to have their in- 
volvement and knowledge from the start. It is also important to have higher- 
level involvement to approve the system. This should be done as the system is 
being developed, not delayed until the system is complete. We believe our 
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technique of getting approval on products and indicators before starting 
contingency development proved very valuable. It not only helped clarify policy 
earlier in the process, but also helped prepare everyone for the eventual 
implementation, This approach gave all levels of the organization a chance to 
learn about the system as it was being developed so that they would know how to 
use it when it was finished. It also served to generate considerable eagerness at 
all levels to receive the first feedback from the system. 

It also proved very effective to have unit personnel who developed the 
system defend it when it was presented to higher management, They were much 
more knowledgeable than the researchers about to? subtleties of unit operation, 
and could better address management's questions and concerns, Also, it gave 
them a chance to present their perceptions of optimal policy. Finally, their sense 
of ownership of the system was increased by their defense of it, 

We also learned toe importance of using a multiple meeting structure that 
allows for an iterative approach to the development of the system. Our strategy 
was to summarize toe results of each meeting and present them at the next 
meeting to assure consensus was reached, In this way, unit personnel had time 
to think about toe issues, discuss them among themselves, and be prepared to ap- 
proach the question with fresh perspectives at the next meeting. Personnel need 
time to adjust to toe idea of completely capturing what they do in a productivity 
measurement system. They need to think about it, consider its implications, and 
be able to discuss it among themselves thoroughly, We believe that a quality 
system could not have been built without multiple meetings separated in time. 

Care must be used during development to ensure that the resulting indica- 
tors are measures which are under the control of toe unit (Hurst s 1980), The 
researchers needed to frequently remind unit personnel to assess whether they 
had control, over a given indicator. Unless toe unit has full control over the 
work being performed, including measures of performance on that work would 
be counterproductive in that they would decrease the motivational impact of the 
system. 

Finally, it is important that the researchers develop good rapport with the 
operational personneL Some personnel were initially suspicious of our intentions, 
and had questions about our credibility, Their experience had been that outsiders, 
who were not experts on their operations, had sometimes imposed programs that 
at best resulted in extra work, and at worst had a negative impact on their 
effectiveness. Taking toe time to really learn what they did, and getting to 
know them, went a long way toward decreasing these barriers. 
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Other Appli cations Of T his Approach 



The productivity measurement system used in this research has a number 
of possible applications. We have discussed how the system can be aggregated to 
larger and larger units. In theory, the system could be developed for all units of 
a very large organization. There would be an overall effectiveness index for 
the entire organization, and for each subunit down to the lowest level in the or- 
ganization. It would be theoretically possible, for example, to develop such a 
system for the entire Air Force. Although such a project would take a tremen- 
dous amount of effort, it is feasible. 

svst^f^ i ^ c a f pl ^ ation is in *e area of management information 
systems (MS . An MIS is designed to provide information to high-level decision 
makers for planning and resource allocation purposes. The problem with the in- 
formation m an MS is that there is frequently too much of it. It is hard to 
separate out the crucial from the not so crucial. It is particularly hard to com- 
bine the information in a way that will facilitate decision making. One appliea- 
T n ,^ c 0l L r approach would be to coia Wae the individual pieces of information in 
the MS by using the contingency approach. For example, if there were ten 
Pieces of information about the functioning of a given unit, developing contingen- 
cies relating these ten measures to overall functioning would make the 
interpretation of how well that part of the organization was doing much easier. 
It would also make comparisons of that unit to other units easier since each 
would be measured on the same metric. 

Another possible application of this approach is in the area of criterion 
development The multiple criteria issue has been a problem for years 
(Dunnette, 1963; Schmidt & Kaplan, 1971). The issue is that for any job there 
are multiple activities and multiple Indices of performance. There may be rat- 
ings on different dimensions, job samples, and measures of output. The problem 
is how to combine these different indices into a single measure of performance. 

One reason it is desirable to have such a single index is for purposes of 
test validation. Dealing with multiple predictors such as tests and biographical 
data is no particular problem in personnel selection. Multiple regression tech- 
niques have been used successfully for some time. It is much more difficult to 
deal with multiple criteria. If the different criterion measures could be com- 
bined into a single index of performance, it would be much easier to prove that 
high test scores do indeed predict successful performance on the job. 

Our approach would be to treat the multiple criteria of performance as 
indicators and develop contingencies for them. A set of contingencies would be 
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developed for each type of job, This preserves the relative importance of the 
different criteria and also preserves non-linearities, A single index of perfor- 
mance could then be developed for use as a criterion. 

Another aspect of criterion development that deserves special attention is 
criteria of managerial performance, Developing good managerial criteria has al- 
ways been a difficult task. Our system could be of use here. We assume that 
the most important task of the manager is to use the resources under his/her 
control to meet the objectives of the unit in the most effective manner possible. 
This implies that one of the most important aspects of the performance of the 
manager is the productivity of the units under his/her control. Thus, developing 
the productivity measurement system for a set of units in the organization also 
creates a criterion for the performance of the units* managers, 

The approach can also be used as the basis for developing performance 
appraisal systems. As with the criterion in selection, one of the problems in 
performance appraisal has been the lack of an overall index of performance, 
Typically ratings of performance have been made on performance dimensions, 
and an overall evaluation made as a separate global rating* or as some summation 
of the ratings of the individual dimensions. 

We would apply the logic of the productivity measurement system to de- 
velop the appraisal system, Once the dimensions were identified, anchors for 
each scale would be developed that were analogous to indicators. That is, they 
would represent behavioral indicators of how well the person was doing on that 
dimension. These would then be used to develop contingencies. Once this was 
done, the overall performance rating would then be determined by summing the 
effectiveness scores for each dimension in a fashion similar to the way in which 
overall effectiveness is determined in the productivity measurement system. 

This would have several advantages. First, an overall measure of per- 
formance would be generated, It would weight the dimensions of performance 
according to their importance, and preserve non-linearities. Second, by keeping 
the ratings to the more molecular dimension ratings, the rater would not have to 
make the more molar judgment about overall performance. This overall judg- 
ment would be generated mechanically. This could have the effect of decreasing 
rater errors. Next, the very process of developing the system should help in 
role clarification, as it does when used as a productivity measurement system, 
Finally, such a system would be useful for performance feedback, The system 
itself would communicate what is important and what is less important, and what 
level of performance is expected in each area. The ratings from the system 
would indicate what the person did well on f and not so well on* It would give an 
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APPENDIX B. MS&D FEEDBACK REPORTS 



PRODUCTIVITY REPORT 
MS&D BRANCH 
INDICATOR AND EFFECTIVENESS DATA FOR MAY, 1986 

BRANCH SUMMARY DATA 



INDICATOR 
DATA 


EFFECTIVE- 
NESS 
SCORE 


PERCENT 
OF MAX 
POSSIBLE 


RECEIVING TOTAL 


423 


94% 


STORAGE 8b ISSUE TOTAL 


531 


91% 


PICKUP & DELIVERY TOTAL 


387 


65% 


INSPECTION TOTAL 


631 


87% 


DELINQUENT DOCUMENTS 58 


65 


76% 


BRANCH GRAND TOTAL 


2037 


83% 
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APPENDIX B. (Cont.) 



PRODUCTIVITY REPORT 
MAY, 1986 
RECEIVING SECTION 



DIDICATOR AND EFFECTIVENESS DATA 



INDICATOR 
DATA 



EFFECTIVE- 
NESS 
SCORE 



PERCENT 
OF MAX 
POSSIBLE 



RECEIVE MATERIAL 

% IN-CHECKING ERRORS 



.00 



DISTMBUTE MATERIAL 

PRIOR. 2 DELIVERY TIME (MIN) 15.60 

PRIOR, 4 DELIVERY TIME (HRS) 2.57 

REFUSALS IN RECEIVING 0.00 

MONITOR REJECTS 

CLEARED DELINQUENT REJECTS 0.25 

TOTAL EFFECTIVENESS 



99 



78 
87 
75 



84 
423 



94% 



EFFECTIVE- 
BRANCH LEVEL INDICATOR INDICATOR NESS 

DATA SCORE 

DELINQUENT DOCUMENTS 58 65 



RFFBCTIVBNESS CHANGE FROM LAST MONTH 



LAST THIS 
MONTH MONTH CHANGE 



RECEIVE MATERIAL ,„ OQ 

% IN-CHECKING ERRORS 99 99 



DISTRIBUTE MATERIAL _ e _ 

PRIOR, 2 DELIVERY TIME (MIN) 80 78 -2 

PRIOR, 4 DELIVERY TIME (HRS) 85 87 2 

WHSE REFUSALS IN RECEIVING 75 75 0 



MONITOR REJECTS . 

CLEARED DELINQUENT REJECTS 85 84 -1 
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APPENDIX B, (Cont.) 



RECEIVING SECTION 
POTEWTTAT, EFFECTIVENESS GATW 



INDICATOR 
DATA 
FROM 



RECEIVE MATERIAL 

% IN-CHECKING ERRORS 

DISTKIBIJTB MATERIAL 

PRIOR. 2 DELIVERY TIME (MIN) 
4 DELIVERY TIME (HRS) 
WH5B REFUSALS IN RECEIVING 

MONITOR REJECTS 

CLEARED DELINQUENT REJECTS 



16 
3 
0 



INDICATOR EFFECTIVE- 
DATA NESS 
IS GAIN 



0 
0 
0 



22 

3 

0 
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iPPBNDLX B. (Cont.) 

FHQDUCTIVITY EEIEPQRT: MAY, 1986 
STORAGE AND ISSUE SECTION 

INDICATO R AND E PPECTIVEHES S CTATA 



EFFECTIVE- 

INEOICATOR NESS 

DATA SCORE 

STORE ITEMS JB WREHOTJSE 

PROPER LOCATION:! REFUSALS 0 85 

PROPER PROCEDURE: # FINDINGS 18 4 

WAREHOUSE MAINTENANCE 

MAINTAIN LOCATIONS OFF R36 98.13 83 

ISSUE REQUESTED ITEMS 

PRIOR, 2 DELIVERY TIME (MIN) 9.96 100 

PRIOR, 3 DELIVERY TIME (MIN) 17.40 99 

PRIOR. 4 DELIVERY TIME (HRS) 1.39 90 

RESPOND TO STOVILLAHCES 

# REPEAT FINDINGS 0 70 

TOTAL EFFECTWEJilSS 531 

EFFECTIVE 

BRANCH LEVEL INDICATOR INDICATOR NESS 

DATA SCORE 

DELINQUENT DOCUMENTS 58 65 



EFFECTrVENESS CHARGE PROM MLAST MONTH 

LAST THIS 

mmwm MONTH 



PROPER LOCATION: I REFUSALS 
PROPER PROCEDURE 0 FINDINGS 

MAINTAIN LOCATION OFF R36 

PRIOR 2 DELIVERY TIME (MIN) 
PRIOR, 3 DELIVERY TIME (MIN) 
PRIOR, 4 DELIVERY TIME (HRS) 

# REPEAT FINDINGS 



85 
34 



85 
4 



87 

100 

98 

88 



83 

100 

99 

90 

70 
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APPENDIX B. (Cont.) 



STORAGE AND ISSUE SECTION 



POTENTIA L EFFECTIVENESS GADJ 

. EFFECTIVE- 
INDICATOR INDICATOR NESS 
FROM TO GAIN 

STORE ITfcMS IN WAREHOUSE 

PROPER LOCATION: #REFUSALS 0 0 n 

PROPER PROCEDURE: # FINDINGS 18 14 g 

WAREHOUSE MAINTENANCE 

MAINTAIN LOCATION: % OFF R36 98 100 9 
ISSUE REQUESTED ITEMS 

PRIOR 2 DELIVERY TIME (MINI 10 0 n 

PRIOR. 3 DELIVERY TIME (MIN) 17 4 q 



0 



PRIOR. 4 DELIVERY TIME (HRS) 1 0 
RESPOND TO SURVEmLANCES 

# REPEAT FINDINGS 0 0 0 
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APPENDIX B, (Cont.) 



PRODUCTIVITY REPORT: MAY, 1986 
PICKUP & DELIVERY SECTION 



INDICATOR 
DATA 



PICK UP U KN-1NS 

# PELlNQCJENT TURN-INS 



DELIVER I JEMS PROPERLY 

PRIOR, 2 I DELIVERY TIME (MIN) 27.60 

PRIOR 2 TSm DELIVERY TIME (MIN) 34,20 

PRIOR. 3 D»ELIVERY TIME (MIN) 39,00 

PRIOR 4 PsELTVERY TIME (HRS) 15.34 

# DELrVEF^ED WRONG LOCATION 0 

MAINTAIN VEHICLES 

VEHICLE INSPECTION SCORE 90.00 

# EEPORT_ABLE ACCIDENTS 0 

# NONREFsDRTABLE ACCIDENTS 0 



EFFECTIVE- 
NESS 
SCORE 



65 



41 
68 
46 
27 
50 



15 
50 
25 



PfiSQCENT 
O^ttMAX 
POSSIBLE 



TOTAL EFFECTIVENESS FOR MONTH 387 65 5% 

EFFECTIVE- 
BRANCH LBVEL INDICATOR INDICATOR NESS 

DATA SCORE 

DELINQUEfti T DOCUMENTS 58 65 



EPFBCTIVIgNeSS CHANGE FROM LAST MONTH 



# DELINQUENT TURN-INS 

PRIOR. 2 I PZ32LIVERY TIME (MIN) 
PRIOR 2 D DELIVERY TIME (MIN) 
PRIOR, 3 QE5LIVERY TIME (MIN) 
PRIOR 4 DESLTVERY TIME (HRS) 

# DELIVBRE-D WRONG LOCATION 

VEHICLE IfT^SPECTION SCORE 

# REPORTABLE ACCIDENTS 

# NONREPOF=*TABLE ACCIDENTS 



LAST THIS 

MONTH MONTH QHA>^JGE 

65 65 0 

66 41 -25 
40 68 2g 
55 46 -9 
27 27 0 
50 50 0 

30 15 -IB 

50 50 0 

25 25 0 
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APPENDIX B, (ContO 



PICKUP & DELIVERY SECTION 



POTENTIAL EFFECTTTOTfTOg^^ 

INDICATOR ifTDlC^TO 
PICK UP TURN-INS IS 

# DELINQUENT TURN-INS Q D 
DELIVER Timm PROPERLY 

S5J2S" 2 1 DELIVERY TIME (MINI 28 A 

PRIOR. 2D DELIVERY TIME (MINI 34 I 

PRIOR 3 DELIVERY TIME (MW) " ft t 

PRIOR 4 DELIVERY TIME (HRS) 15 n 

# DELIVERED WRONG LOCATION 0 {j 

MADITAnr VEinCLES 

Ji3i%S, INSPECTION SCORE 90 9a 

# REPORTABLE ACCIDENTS n •? 

# NGNREPORTABLE ACCIDENTS 0 Q 



APPENDIX B. (Cont.) 



PRODUCTIVITY REPORT: MAY, 1986 
INSPECTION SECTION 
nroiCATOR AMD EFFEO'nVENESS DATA 



EFFECTIVE- PERCENT 
INDICATOR NESS OF MAX 

DATA SCORE POSSIBLE 



AVG # LOCAL PURCHASE ITEMS 

LEFT TO BE INSPECTED 2.84 69 

AVG # INCQMMING DATED ITEMS 

LEFT TO BE INSPECTED 0.00 78 

AVG # ITEMS W/O IDENTIFICATION 

LEFT TO BE INSPECTED 5.16 19 



AVG # TURNED-tN AIRCRAFT 

PARTS LEFT TO BE INSPECTED 0.11 91 

AVG # FUNCTIONAL CHECKS 

LEFT TO BE COORDINATED 0.11 83 

AVG # SUSPECT ITEMS 

LEFT TO BE INSPECTED 0.00 93 

# LATE MONTHLY INSPECTIONS 0.00 15 

INSPECT DPDO MATERIAL (# RET.) 0.00 15 

TECH, ORDERS} % TCTOs CHECKED 100 85 

SHIPMENTS- # RODS 6.00 4 

SHIPMENTS: AVG # SHIPMENTS 

LEFT TO BE INSPECTED 0,16 79 

TOTAL EFFECTIVENESS 631 87% 

BRANCH LEVEL INDICATOR INDICATOR EFFECTIVENESS 

DATA SCORE 

DELINQUENT DOCUMENTS 58 65 
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APPENDIX B. (Cont.) 



INSPECTION SECTION 



BPFBCTTFEMESS CHANGE FROM LAST MONTH 



AVG # LOCAL PURCHASE ITEMS 
LEFT TO BE INSPECTED 

AVG # INCOMMING DATED ITEMS 
LEFT TO BE INSPECTED 



LAST 
MONTH 



-6 
78 



AVG # ITEMS W/O IDENTIFICATION 
LEFT TO BE INSPECTED 20 

AVG # TURNED-IN AIRCRAFT PARTS 
LEFT TO BE INSPECTED 98 

AVG § FUNCTIONAL CHECKS 

LEFT TO BE COORDINATED -17 

AVG # SUSPECT ITEMS 

LEFT TO BE INSPECTED 93 

# LATE MONTHLY INSPECTIONS 15 

INSPECT DPDO MATERIAL (# RET.) 15 

TECH ORDERS: % TCTOs CHECKED 85 

SHIPMENTS; # RODS 



SHIPMENTS: AVG # SHIPMENTS 
LEFT TO BE INSPECTED 



-10 
83 



THIS 
MONTH 



69 

78 

19 

91 

83 

93 
IS 
15 
85 
4 

79 



CHANGE 

75 

0 

-1 

-7 

100 

0 
0 
0 
0 

14 

-4 
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APPENDIX B. (Concluded) 

INSPECTION SECTION 
POTENTIAL EFFECTIVENESS GAIN 

INDICATOR INDICATOR 



PROM TO 

AVG # LOCAL ^PURCHASE ITEMS 

LEFT TO BE INSPECTED 3 0 

AVG it INCOMMING DATED ITEMS 

LEFT TO BE INSPECTED 0 0 

AVG # ITEMS W/O IDENTIFICATION 

LEFT TO BE INSPECTED 5 0 

AVG # TURNED-IN AIRCRAFT PARTS 

LEFT TO BE INSPECTED 0 0 

AVG # FUNCTIONAL CHECKS 

LEFT TO BE COORDINATED 0 0 

AVG # SUSPECT ITEMS 

LEFT TO BE INSPECTED 0 0 

# LATE MONTHLY INSPECTIONS 0 0 

INSPECT DPDO MATERIAL (# RET,} 0 0 

TECH. ORDERS: % TCTOs CHECKED 100 100 

SHIPMENTS: # RODS 6 0 

SHIPMENTS: AVG # SHIPMENTS 

LEFT TO BE INSPECTED 0 0 
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